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Preface 


Pattern recognition has its origins in engineering, whereas machine learning grew 
out of computer science. However, these activities can be viewed as two facets of 
the same field, and together they have undergone substantial development over the 
past ten years. In particular, Bayesian methods have grown from a specialist niche to 
become mainstream, while graphical models have emerged as a general framework 
for describing and applying probabilistic models. Also, the practical applicability of 
Bayesian methods has been greatly enhanced through the development of a range of 
approximate inference algorithms such as variational Bayes and expectation propa- 
gation. Similarly, new models based on kernels have had significant impact on both 
algorithms and applications. 

This new textbook reflects these recent developments while providing a compre- 
hensive introduction to the fields of pattern recognition and machine learning. It is 
aimed at advanced undergraduates or first year PhD students, as well as researchers 
and practitioners, and assumes no previous knowledge of pattern recognition or ma- 
chine learning concepts. Knowledge of multivariate calculus and basic linear algebra 
is required, and some familiarity with probabilities would be helpful though not es- 
sential as the book includes a self-contained introduction to basic probability theory. 

Because this book has broad scope, it is impossible to provide a complete list of 
references, and in particular no attempt has been made to provide accurate historical 
attribution of ideas. Instead, the aim has been to give references that offer greater 
detail than is possible here and that hopefully provide entry points into what, in some 
cases, is a very extensive literature. For this reason, the references are often to more 
recent textbooks and review articles rather than to original sources. 

The book is supported by a great deal of additional material, including lecture 
slides as well as the complete set of figures used in the book, and the reader is 
encouraged to visit the book web site for the latest information: 


http://research.microsoft.com/~cmbishop/PRML 
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Exercises 


The exercises that appear at the end of every chapter form an important com- 
ponent of the book. Each exercise has been carefully chosen to reinforce concepts 
explained in the text or to develop and generalize them in significant ways, and each 
is graded according to difficulty ranging from (x), which denotes a simple exercise 
taking a few minutes to complete, through to (« x x), which denotes a significantly 
more complex exercise. 

It has been difficult to know to what extent these solutions should be made 
widely available. Those engaged in self study will find worked solutions very ben- 
eficial, whereas many course tutors request that solutions be available only via the 
publisher so that the exercises may be used in class. In order to try to meet these 
conflicting requirements, those exercises that help amplify key points in the text, or 
that fill in important details, have solutions that are available as a PDF file from the 
book web site. Such exercises are denoted by RNW. Solutions for the remaining 
exercises are available to course tutors by contacting the publisher (contact details 
are given on the book web site). Readers are strongly encouraged to work through 
the exercises unaided, and to turn to the solutions only as required. 

Although this book focuses on concepts and principles, in a taught course the 
students should ideally have the opportunity to experiment with some of the key 
algorithms using appropriate data sets. A companion volume (Bishop and Nabney, 
2008) will deal with practical aspects of pattern recognition and machine learning, 
and will be accompanied by Matlab software implementing most of the algorithms 
discussed in this book. 
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Mathematical notation 


I have tried to keep the mathematical content of the book to the minimum neces- 
sary to achieve a proper understanding of the field. However, this minimum level is 
nonzero, and it should be emphasized that a good grasp of calculus, linear algebra, 
and probability theory is essential for a clear understanding of modern pattern recog- 
nition and machine learning techniques. Nevertheless, the emphasis in this book is 
on conveying the underlying concepts rather than on mathematical rigour. 

I have tried to use a consistent notation throughout the book, although at times 
this means departing from some of the conventions used in the corresponding re- 
search literature. Vectors are denoted by lower case bold Roman letters such as 
x, and all vectors are assumed to be column vectors. A superscript T denotes the 
transpose of a matrix or vector, so that xT will be a row vector. Uppercase bold 


roman letters, such as M, denote matrices. The notation (w1,..., waz) denotes a 

row vector with M elements, while the corresponding column vector is written as 
= T 

w = (w1,..., WM)". 


The notation [a,b] is used to denote the closed interval from a to b, that is the 
interval including the values a and b themselves, while (a, b) denotes the correspond- 
ing open interval, that is the interval excluding a and b. Similarly, [a, b) denotes an 
interval that includes a but excludes b. For the most part, however, there will be 
little need to dwell on such refinements as whether the end points of an interval are 
included or not. 

The M x M identity matrix (also known as the unit matrix) is denoted Iw, 
which will be abbreviated to I where there is no ambiguity about it dimensionality. 
It has elements J;; that equal 1 if i = j and 0 if i Æ j. 

A functional is denoted f[y] where y(x) is some function. The concept of a 
functional is discussed in Appendix D. 

The notation g(x) = O(f(«)) denotes that | f(a) /g(a)| is bounded as x — oo. 
For instance if g(a) = 3x7 + 2, then g(x) = O(a’). 

The expectation of a function f(x,y) with respect to a random variable x is de- 
noted by E,,[ f(x, y)]. In situations where there is no ambiguity as to which variable 
is being averaged over, this will be simplified by omitting the suffix, for instance 


xi 
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MATHEMATICAL NOTATION 


Efx]. If the distribution of x is conditioned on another variable z, then the corre- 
sponding conditional expectation will be written E,[f(2)|z]. Similarly, the variance 
is denoted var[ f (x)], and for vector variables the covariance is written cov|x, y]. We 
shall also use cov[x] as a shorthand notation for cov[x, x]. The concepts of expecta- 
tions and covariances are introduced in Section 1.2.2. 

If we have N values x,,...,Xy of a D-dimensional vector x = (21,...,2p)*, 
we can combine the observations into a data matrix X in which the n‘ row of X 
corresponds to the row vector x]. Thus the n,i element of X corresponds to the 
it! element of the n*" observation xn. For the case of one-dimensional variables we 
shall denote such a matrix by x, which is a column vector whose nt? element is £n. 
Note that x (which has dimensionality NV) uses a different typeface to distinguish it 
from x (which has dimensionality D). 
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The problem of searching for patterns in data is a fundamental one and has a long and 
successful history. For instance, the extensive astronomical observations of Tycho 
Brahe in the 16' century allowed Johannes Kepler to discover the empirical laws of 
planetary motion, which in turn provided a springboard for the development of clas- 
sical mechanics. Similarly, the discovery of regularities in atomic spectra played a 
key role in the development and verification of quantum physics in the early twenti- 
eth century. The field of pattern recognition is concerned with the automatic discov- 
ery of regularities in data through the use of computer algorithms and with the use of 
these regularities to take actions such as classifying the data into different categories. 

Consider the example of recognizing handwritten digits, illustrated in Figure 1.1. 
Each digit corresponds to a 28 x 28 pixel image and so can be represented by a vector 
x comprising 784 real numbers. The goal is to build a machine that will take such a 
vector x as input and that will produce the identity of the digit 0,...,9 as the output. 
This is a nontrivial problem due to the wide variability of handwriting. It could be 
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Figure 1.1 


Examples of hand-written dig- 


its taken from US zip codes. Ò / 7 ay 4 
s]lellzllella 


tackled using handcrafted rules or heuristics for distinguishing the digits based on 
the shapes of the strokes, but in practice such an approach leads to a proliferation of 
rules and of exceptions to the rules and so on, and invariably gives poor results. 

Far better results can be obtained by adopting a machine learning approach in 
which a large set of N digits {x1,..., Xy} called a training set is used to tune the 
parameters of an adaptive model. The categories of the digits in the training set 
are known in advance, typically by inspecting them individually and hand-labelling 
them. We can express the category of a digit using target vector t, which represents 
the identity of the corresponding digit. Suitable techniques for representing cate- 
gories in terms of vectors will be discussed later. Note that there is one such target 
vector t for each digit image x. 

The result of running the machine learning algorithm can be expressed as a 
function y(x) which takes a new digit image x as input and that generates an output 
vector y, encoded in the same way as the target vectors. The precise form of the 
function y(x) is determined during the training phase, also known as the learning 
phase, on the basis of the training data. Once the model is trained it can then de- 
termine the identity of new digit images, which are said to comprise a test set. The 
ability to categorize correctly new examples that differ from those used for train- 
ing is known as generalization. In practical applications, the variability of the input 
vectors will be such that the training data can comprise only a tiny fraction of all 
possible input vectors, and so generalization is a central goal in pattern recognition. 

For most practical applications, the original input variables are typically prepro- 
cessed to transform them into some new space of variables where, it is hoped, the 
pattern recognition problem will be easier to solve. For instance, in the digit recogni- 
tion problem, the images of the digits are typically translated and scaled so that each 
digit is contained within a box of a fixed size. This greatly reduces the variability 
within each digit class, because the location and scale of all the digits are now the 
same, which makes it much easier for a subsequent pattern recognition algorithm 
to distinguish between the different classes. This pre-processing stage is sometimes 
also called feature extraction. Note that new test data must be pre-processed using 
the same steps as the training data. 

Pre-processing might also be performed in order to speed up computation. For 
example, if the goal is real-time face detection in a high-resolution video stream, 
the computer must handle huge numbers of pixels per second, and presenting these 
directly to a complex pattern recognition algorithm may be computationally infeasi- 
ble. Instead, the aim is to find useful features that are fast to compute, and yet that 
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also preserve useful discriminatory information enabling faces to be distinguished 
from non-faces. These features are then used as the inputs to the pattern recognition 
algorithm. For instance, the average value of the image intensity over a rectangular 
subregion can be evaluated extremely efficiently (Viola and Jones, 2004), and a set of 
such features can prove very effective in fast face detection. Because the number of 
such features is smaller than the number of pixels, this kind of pre-processing repre- 
sents a form of dimensionality reduction. Care must be taken during pre-processing 
because often information is discarded, and if this information is important to the 
solution of the problem then the overall accuracy of the system can suffer. 

Applications in which the training data comprises examples of the input vectors 
along with their corresponding target vectors are known as supervised learning prob- 
lems. Cases such as the digit recognition example, in which the aim is to assign each 
input vector to one of a finite number of discrete categories, are called classification 
problems. If the desired output consists of one or more continuous variables, then 
the task is called regression. An example of a regression problem would be the pre- 
diction of the yield in a chemical manufacturing process in which the inputs consist 
of the concentrations of reactants, the temperature, and the pressure. 

In other pattern recognition problems, the training data consists of a set of input 
vectors x without any corresponding target values. The goal in such unsupervised 
learning problems may be to discover groups of similar examples within the data, 
where it is called clustering, or to determine the distribution of data within the input 
space, known as density estimation, or to project the data from a high-dimensional 
space down to two or three dimensions for the purpose of visualization. 

Finally, the technique of reinforcement learning (Sutton and Barto, 1998) is con- 
cerned with the problem of finding suitable actions to take in a given situation in 
order to maximize a reward. Here the learning algorithm is not given examples of 
optimal outputs, in contrast to supervised learning, but must instead discover them 
by a process of trial and error. Typically there is a sequence of states and actions in 
which the learning algorithm is interacting with its environment. In many cases, the 
current action not only affects the immediate reward but also has an impact on the re- 
ward at all subsequent time steps. For example, by using appropriate reinforcement 
learning techniques a neural network can learn to play the game of backgammon to a 
high standard (Tesauro, 1994). Here the network must learn to take a board position 
as input, along with the result of a dice throw, and produce a strong move as the 
output. This is done by having the network play against a copy of itself for perhaps a 
million games. A major challenge is that a game of backgammon can involve dozens 
of moves, and yet it is only at the end of the game that the reward, in the form of 
victory, is achieved. The reward must then be attributed appropriately to all of the 
moves that led to it, even though some moves will have been good ones and others 
less so. This is an example of a credit assignment problem. A general feature of re- 
inforcement learning is the trade-off between exploration, in which the system tries 
out new kinds of actions to see how effective they are, and exploitation, in which 
the system makes use of actions that are known to yield a high reward. Too strong 
a focus on either exploration or exploitation will yield poor results. Reinforcement 
learning continues to be an active area of machine learning research. However, a 
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Figure 1.2 Plot of a training data set of N = 


1.1. 


10 points, shown as blue circles, 
each comprising an observation 

of the input variable x along with 
the corresponding target variable + 
t. The green curve shows the 
function sin(27x) used to gener- 
ate the data. Our goal is to pre- OF 
dict the value of t for some new 
value of xz, without knowledge of 
the green curve. 


detailed treatment lies beyond the scope of this book. 

Although each of these tasks needs its own tools and techniques, many of the 
key ideas that underpin them are common to all such problems. One of the main 
goals of this chapter is to introduce, in a relatively informal way, several of the most 
important of these concepts and to illustrate them using simple examples. Later in 
the book we shall see these same ideas re-emerge in the context of more sophisti- 
cated models that are applicable to real-world pattern recognition applications. This 
chapter also provides a self-contained introduction to three important tools that will 
be used throughout the book, namely probability theory, decision theory, and infor- 
mation theory. Although these might sound like daunting topics, they are in fact 
straightforward, and a clear understanding of them is essential if machine learning 
techniques are to be used to best effect in practical applications. 


Example: Polynomial Curve Fitting 


We begin by introducing a simple regression problem, which we shall use as a run- 
ning example throughout this chapter to motivate a number of key concepts. Sup- 
pose we observe a real-valued input variable x and we wish to use this observation to 
predict the value of a real-valued target variable t. For the present purposes, it is in- 
structive to consider an artificial example using synthetically generated data because 
we then know the precise process that generated the data for comparison against any 
learned model. The data for this example is generated from the function sin(272) 
with random noise included in the target values, as described in detail in Appendix A. 

Now suppose that we are given a training set comprising N observations of zx, 
written X = (x1,...,2N)", together with corresponding observations of the values 
of t, denoted t = (t,,..., ty)". Figure 1.2 shows a plot of a training set comprising 
N = 10 data points. The input data set X in Figure 1.2 was generated by choos- 
ing values of zn, for n = 1,..., N, spaced uniformly in range [0, 1], and the target 
data set t was obtained by first computing the corresponding values of the function 
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sin(272) and then adding a small level of random noise having a Gaussian distri- 
bution (the Gaussian distribution is discussed in Section 1.2.4) to each such point in 
order to obtain the corresponding value tn. By generating data in this way, we are 
capturing a property of many real data sets, namely that they possess an underlying 
regularity, which we wish to learn, but that individual observations are corrupted by 
random noise. This noise might arise from intrinsically stochastic (i.e. random) pro- 
cesses such as radioactive decay but more typically is due to there being sources of 
variability that are themselves unobserved. 

Our goal is to exploit this training set in order to make predictions of the value 
T of the target variable for some new value Ẹ of the input variable. As we shall see 
later, this involves implicitly trying to discover the underlying function sin(272). 
This is intrinsically a difficult problem as we have to generalize from a finite data 
set. Furthermore the observed data are corrupted with noise, and so for a given © 
there is uncertainty as to the appropriate value for t. Probability theory, discussed 
in Section 1.2, provides a framework for expressing such uncertainty in a precise 
and quantitative manner, and decision theory, discussed in Section 1.5, allows us to 
exploit this probabilistic representation in order to make predictions that are optimal 
according to appropriate criteria. 

For the moment, however, we shall proceed rather informally and consider a 
simple approach based on curve fitting. In particular, we shall fit the data using a 
polynomial function of the form 


M 
M j 
ylz, w) = wo + wiz + wz +... + wmr” = ) wjr? (1.1) 
j=0 


where M is the order of the polynomial, and z denotes x raised to the power of j. 
The polynomial coefficients wo,..., wag are collectively denoted by the vector w. 
Note that, although the polynomial function y(x, w) is a nonlinear function of x, it 
is a linear function of the coefficients w. Functions, such as the polynomial, which 
are linear in the unknown parameters have important properties and are called linear 
models and will be discussed extensively in Chapters 3 and 4. 

The values of the coefficients will be determined by fitting the polynomial to the 
training data. This can be done by minimizing an error function that measures the 
misfit between the function y(x, w), for any given value of w, and the training set 
data points. One simple choice of error function, which is widely used, is given by 
the sum of the squares of the errors between the predictions y(«,, w) for each data 
point £n and the corresponding target values tn, so that we minimize 


ioe g 
E(w) = 3) {y(en,w) — tn} (1.2) 


where the factor of 1/2 is included for later convenience. We shall discuss the mo- 
tivation for this choice of error function later in this chapter. For the moment we 
simply note that it is a nonnegative quantity that would be zero if, and only if, the 
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Figure 1.3 The error function (1.2) corre- 


Exercise 1.1 


sponds to (one half of) the sum of + tn 
the squares of the displacements 
(shown by the vertical green bars) 
of each data point from the function 
y(x,w). 
Y(2n, W) 


In 


function y(x, w) were to pass exactly through each training data point. The geomet- 
rical interpretation of the sum-of-squares error function is illustrated in Figure 1.3. 


We can solve the curve fitting problem by choosing the value of w for which 
E(w) is as small as possible. Because the error function is a quadratic function of 
the coefficients w, its derivatives with respect to the coefficients will be linear in the 
elements of w, and so the minimization of the error function has a unique solution, 
denoted by w*, which can be found in closed form. The resulting polynomial is 
given by the function y(x, w*). 

There remains the problem of choosing the order M of the polynomial, and as 
we shall see this will turn out to be an example of an important concept called model 
comparison or model selection. In Figure 1.4, we show four examples of the results 
of fitting polynomials having orders M = 0,1,3, and 9 to the data set shown in 
Figure 1.2. 

We notice that the constant (M = 0) and first order (M = 1) polynomials 
give rather poor fits to the data and consequently rather poor representations of the 
function sin(272). The third order (M = 3) polynomial seems to give the best fit 
to the function sin(272) of the examples shown in Figure 1.4. When we go to a 
much higher order polynomial (M = 9), we obtain an excellent fit to the training 
data. In fact, the polynomial passes exactly through each data point and E(w*) = 0. 
However, the fitted curve oscillates wildly and gives a very poor representation of 
the function sin(272). This latter behaviour is known as over-fitting. 

As we have noted earlier, the goal is to achieve good generalization by making 
accurate predictions for new data. We can obtain some quantitative insight into the 
dependence of the generalization performance on M by considering a separate test 
set comprising 100 data points generated using exactly the same procedure used 
to generate the training set points but with new choices for the random noise values 
included in the target values. For each choice of M, we can then evaluate the residual 
value of E(w*) given by (1.2) for the training data, and we can also evaluate E(w”) 
for the test data set. It is sometimes more convenient to use the root-mean-square 
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T T 


Figure 1.4 Plots of polynomials having various orders M, shown as red curves, fitted to the data set shown in 
Figure 1.2. 


(RMS) error defined by 
Eros = \/ 2E(w*)/N (1.3) 


in which the division by N allows us to compare different sizes of data sets on 
an equal footing, and the square root ensures that Erms is measured on the same 
scale (and in the same units) as the target variable t. Graphs of the training and 
test set RMS errors are shown, for various values of M, in Figure 1.5. The test 
set error is a measure of how well we are doing in predicting the values of t for 
new data observations of x. We note from Figure 1.5 that small values of M give 
relatively large values of the test set error, and this can be attributed to the fact that 
the corresponding polynomials are rather inflexible and are incapable of capturing 
the oscillations in the function sin(27x). Values of M in the range 3 < M < 8 
give small values for the test set error, and these also give reasonable representations 
of the generating function sin(27x), as can be seen, for the case of M = 3, from 
Figure 1.4. 
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Figure 1.5 Graphs of the root-mean-square 

error, defined by (1.3), evaluated 1 : — 
on the training set and on an inde- —6— Training 
pendent test set for various values —sé— Test 
of M. 

3 

g 0.5 

A 
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Table 1.1 


For M = 9, the training set error goes to zero, as we might expect because 
this polynomial contains 10 degrees of freedom corresponding to the 10 coefficients 
Wo, ..., Wg, and so can be tuned exactly to the 10 data points in the training set. 
However, the test set error has become very large and, as we saw in Figure 1.4, the 
corresponding function y(x, w*) exhibits wild oscillations. 

This may seem paradoxical because a polynomial of given order contains all 
lower order polynomials as special cases. The M = 9 polynomial is therefore capa- 
ble of generating results at least as good as the M = 3 polynomial. Furthermore, we 
might suppose that the best predictor of new data would be the function sin(272) 
from which the data was generated (and we shall see later that this is indeed the 
case). We know that a power series expansion of the function sin(27x) contains 
terms of all orders, so we might expect that results should improve monotonically as 
we increase M. 

We can gain some insight into the problem by examining the values of the co- 
efficients w* obtained from polynomials of various order, as shown in Table 1.1. 
We see that, as M increases, the magnitude of the coefficients typically gets larger. 
In particular for the M = 9 polynomial, the coefficients have become finely tuned 
to the data by developing large positive and negative values so that the correspond- 


Table of the coefficients w* for M=0 M=1 M=6 M=9 

polynomials of various order. w7 0.19 0.82 0.31 0.35 

Observe how the typical mag- n 

nitude of the coefficients in- %1 -1.27 7.99 232.37 

creases dramatically as the or- w3 -25.43 -5321.83 

der of the polynomial increases. w3 17.37 48568.31 
wi -231639.30 
we 640042.26 
we -1061800.52 
wr 1042400.18 
wR -557682.99 
We 125201.43 
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Figure 1.6 Plots of the solutions obtained by minimizing the sum-of-squares error function using the M = 9 
polynomial for N = 15 data points (left plot) and N = 100 data points (right plot). We see that increasing the 
size of the data set reduces the over-fitting problem. 


Section 3.4 


ing polynomial function matches each of the data points exactly, but between data 
points (particularly near the ends of the range) the function exhibits the large oscilla- 
tions observed in Figure 1.4. Intuitively, what is happening is that the more flexible 
polynomials with larger values of M are becoming increasingly tuned to the random 
noise on the target values. 

It is also interesting to examine the behaviour of a given model as the size of the 
data set is varied, as shown in Figure 1.6. We see that, for a given model complexity, 
the over-fitting problem become less severe as the size of the data set increases. 
Another way to say this is that the larger the data set, the more complex (in other 
words more flexible) the model that we can afford to fit to the data. One rough 
heuristic that is sometimes advocated is that the number of data points should be 
no less than some multiple (say 5 or 10) of the number of adaptive parameters in 
the model. However, as we shall see in Chapter 3, the number of parameters is not 
necessarily the most appropriate measure of model complexity. 

Also, there is something rather unsatisfying about having to limit the number of 
parameters in a model according to the size of the available training set. It would 
seem more reasonable to choose the complexity of the model according to the com- 
plexity of the problem being solved. We shall see that the least squares approach 
to finding the model parameters represents a specific case of maximum likelihood 
(discussed in Section 1.2.5), and that the over-fitting problem can be understood as 
a general property of maximum likelihood. By adopting a Bayesian approach, the 
over-fitting problem can be avoided. We shall see that there is no difficulty from 
a Bayesian perspective in employing models for which the number of parameters 
greatly exceeds the number of data points. Indeed, in a Bayesian model the effective 
number of parameters adapts automatically to the size of the data set. 

For the moment, however, it is instructive to continue with the current approach 
and to consider how in practice we can apply it to data sets of limited size where we 
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Figure 1.7 Plots of M = 9 polynomials fitted to the data set shown in Figure 1.2 using the regularized error 
function (1.4) for two values of the regularization parameter À corresponding to In A = —18 and In à = 0. The 
case of no regularizer, i.e., A = 0, corresponding to In \ = —oo, is shown at the bottom right of Figure 1.4. 


Exercise 1.2 


may wish to use relatively complex and flexible models. One technique that is often 
used to control the over-fitting phenomenon in such cases is that of regularization, 
which involves adding a penalty term to the error function (1.2) in order to discourage 
the coefficients from reaching large values. The simplest such penalty term takes the 
form of a sum of squares of all of the coefficients, leading to a modified error function 
of the form 


a 1 D 2 À r 
B(w) = 3) {y(en,w) — ta}? + Sill (1.4) 
n=1 


where || w||? = wi w = we +w7+...+w%,, and the coefficient À governs the rel- 


ative importance of the regularization term compared with the sum-of-squares error 
term. Note that often the coefficient wo is omitted from the regularizer because its 
inclusion causes the results to depend on the choice of origin for the target variable 
(Hastie et al., 2001), or it may be included but with its own regularization coefficient 
(we shall discuss this topic in more detail in Section 5.5.1). Again, the error function 
in (1.4) can be minimized exactly in closed form. Techniques such as this are known 
in the statistics literature as shrinkage methods because they reduce the value of the 
coefficients. The particular case of a quadratic regularizer is called ridge regres- 
sion (Hoer! and Kennard, 1970). In the context of neural networks, this approach is 
known as weight decay. 

Figure 1.7 shows the results of fitting the polynomial of order M = 9 to the 
same data set as before but now using the regularized error function given by (1.4). 
We see that, for a value of In A = —18, the over-fitting has been suppressed and we 
now obtain a much closer representation of the underlying function sin(27). If, 
however, we use too large a value for À then we again obtain a poor fit, as shown in 
Figure 1.7 for In A = 0. The corresponding coefficients from the fitted polynomials 
are given in Table 1.2, showing that regularization has the desired effect of reducing 
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Table 1.2 Table of the coefficients w* for M = InA=-oo InA=-18 InA=0 
9 polynomials with various values for we 0.35 0.35 0.13 
the regularization parameter A. Note 9 23237 474 0.05 
that nA = —co corresponds toa “1 ; : mee 
model with no regularization, i.e., to We -5321.83 -0.77 -0.06 
the graph at the bottom right in Fig- w3 48568.31 -31.97 -0.05 
ure 1.4. We see that, as the value of wi -231639.30 -3.89 -0.03 
À increases, the typical magnitude of wt 640042.26 55.28 -0.02 
the coefficients gets smaller. w% | -1061800.52 41.32 -0.01 

ws | 1042400.18 -45.95 -0.00 
We -557682.99 -91.53 0.00 
wò 125201.43 72.68 0.01 
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Figure 1.8 


the magnitude of the coefficients. 

The impact of the regularization term on the generalization error can be seen by 
plotting the value of the RMS error (1.3) for both training and test sets against In A, 
as shown in Figure 1.8. We see that in effect \ now controls the effective complexity 
of the model and hence determines the degree of over-fitting. 

The issue of model complexity is an important one and will be discussed at 
length in Section 1.3. Here we simply note that, if we were trying to solve a practical 
application using this approach of minimizing an error function, we would have to 
find a way to determine a suitable value for the model complexity. The results above 
suggest a simple way of achieving this, namely by taking the available data and 
partitioning it into a training set, used to determine the coefficients w, and a separate 
validation set, also called a hold-out set, used to optimize the model complexity 
(either M or A). In many cases, however, this will prove to be too wasteful of 
valuable training data, and we have to seek more sophisticated approaches. 

So far our discussion of polynomial curve fitting has appealed largely to in- 
tuition. We now seek a more principled approach to solving problems in pattern 
recognition by turning to a discussion of probability theory. As well as providing the 
foundation for nearly all of the subsequent developments in this book, it will also 


Graph of the root-mean-square er- 
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Figure 1.9 


give us some important insights into the concepts we have introduced in the con- 
text of polynomial curve fitting and will allow us to extend these to more complex 
situations. 


Probability Theory 


A key concept in the field of pattern recognition is that of uncertainty. It arises both 
through noise on measurements, as well as through the finite size of data sets. Prob- 
ability theory provides a consistent framework for the quantification and manipula- 
tion of uncertainty and forms one of the central foundations for pattern recognition. 
When combined with decision theory, discussed in Section 1.5, it allows us to make 
optimal predictions given all the information available to us, even though that infor- 
mation may be incomplete or ambiguous. 

We will introduce the basic concepts of probability theory by considering a sim- 
ple example. Imagine we have two boxes, one red and one blue, and in the red box 
we have 2 apples and 6 oranges, and in the blue box we have 3 apples and 1 orange. 
This is illustrated in Figure 1.9. Now suppose we randomly pick one of the boxes 
and from that box we randomly select an item of fruit, and having observed which 
sort of fruit it is we replace it in the box from which it came. We could imagine 
repeating this process many times. Let us suppose that in so doing we pick the red 
box 40% of the time and we pick the blue box 60% of the time, and that when we 
remove an item of fruit from a box we are equally likely to select any of the pieces 
of fruit in the box. 

In this example, the identity of the box that will be chosen is a random variable, 
which we shall denote by B. This random variable can take one of two possible 
values, namely r (corresponding to the red box) or b (corresponding to the blue 
box). Similarly, the identity of the fruit is also a random variable and will be denoted 
by F. It can take either of the values a (for apple) or o (for orange). 

To begin with, we shall define the probability of an event to be the fraction 
of times that event occurs out of the total number of trials, in the limit that the total 
number of trials goes to infinity. Thus the probability of selecting the red box is 4/10 


We use a simple example of two 
coloured boxes each containing fruit 
(apples shown in green and or- 
anges shown in orange) to intro- 
duce the basic ideas of probability. 


1.2. Probability Theory 13 


Figure 1.10 Wecan derive the sum and product rules of probability by 
considering two random variables, X, which takes the values {x;} where 
i = 1,..., M, and Y, which takes the values {y;} where j = 1,..., L. 
In this illustration we have M = 5 and L = 3. If we consider a total 
number N of instances of these variables, then we denote the number 
of instances where X = x; and Y = y; by nij, which is the number of 
points in the corresponding cell of the array. The number of points in 
column 7, corresponding to X = x;, is denoted by c;, and the number of 
points in row j, corresponding to Y = y;, is denoted by rj. 


and the probability of selecting the blue box is 6/10. We write these probabilities 
as p(B = r) = 4/10 and p(B = b) = 6/10. Note that, by definition, probabilities 
must lie in the interval [0, 1]. Also, if the events are mutually exclusive and if they 
include all possible outcomes (for instance, in this example the box must be either 
red or blue), then we see that the probabilities for those events must sum to one. 

We can now ask questions such as: “what is the overall probability that the se- 
lection procedure will pick an apple?”, or “given that we have chosen an orange, 
what is the probability that the box we chose was the blue one?”. We can answer 
questions such as these, and indeed much more complex questions associated with 
problems in pattern recognition, once we have equipped ourselves with the two el- 
ementary rules of probability, known as the sum rule and the product rule. Having 
obtained these rules, we shall then return to our boxes of fruit example. 

In order to derive the rules of probability, consider the slightly more general ex- 
ample shown in Figure 1.10 involving two random variables X and Y (which could 
for instance be the Box and Fruit variables considered above). We shall suppose that 
X can take any of the values x; where i = 1,..., M, and Y can take the values yj 
where j = 1,..., L. Consider a total of N trials in which we sample both of the 
variables X and Y, and let the number of such trials in which X = x; and Y = y; 
be nij. Also, let the number of trials in which X takes the value x; (irrespective 
of the value that Y takes) be denoted by c;, and similarly let the number of trials in 
which Y takes the value y; be denoted by rj. 

The probability that X will take the value x; and Y will take the value y; is 
written p(X = «2;,Y = yj) and is called the joint probability of X = x; and 
Y = yj. It is given by the number of points falling in the cell 7,7 as a fraction of the 
total number of points, and hence 


Nij 
N 
Here we are implicitly considering the limit N — oo. Similarly, the probability that 


X takes the value x; irrespective of the value of Y is written as p(X = x;) and is 
given by the fraction of the total number of points that fall in column 2, so that 


p(X = x; Y = yj) = (1.5) 


Ci 

N 
Because the number of instances in column 7 in Figure 1.10 is just the sum of the 
number of instances in each cell of that column, we have c; = De ni; and therefore, 


p(X = zi) = (1.6) 
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from (1.5) and (1.6), we have 
p(X = zi) =) p(X = £Y = y) (1.7) 


which is the sum rule of probability. Note that p(X = x;) is sometimes called the 
marginal probability, because it is obtained by marginalizing, or summing out, the 
other variables (in this case Y). 

If we consider only those instances for which X = .;, then the fraction of 
such instances for which Y = y; is written p(Y = y;|X = 2;) and is called the 
conditional probability of Y = y; given X = 2;. It is obtained by finding the 
fraction of those points in column 2 that fall in cell 2,7 and hence is given by 

Nij 
pY = y| X =x) = Pa (1.8) 


From (1.5), (1.6), and (1.8), we can then derive the following relationship 
Nij Nij ; Ci 
N Ci N 
pY = y| X = xj)p(X = xj) (1.9) 


p(X £i, Y Yj) 


which is the product rule of probability. 

So far we have been quite careful to make a distinction between a random vari- 
able, such as the box B in the fruit example, and the values that the random variable 
can take, for example r if the box were the red one. Thus the probability that B takes 
the value r is denoted p(B = r). Although this helps to avoid ambiguity, it leads 
to a rather cumbersome notation, and in many cases there will be no need for such 
pedantry. Instead, we may simply write p(B) to denote a distribution over the ran- 
dom variable B, or p(r) to denote the distribution evaluated for the particular value 
r, provided that the interpretation is clear from the context. 

With this more compact notation, we can write the two fundamental rules of 
probability theory in the following form. 


The Rules of Probability 


sum rule p(X) =X (X,Y) (1.10) 
Y 


product rule p(X, Y) = p(Y |X) p(X). (1.11) 


Here p(X, Y) is a joint probability and is verbalized as “the probability of X and 
Y”. Similarly, the quantity p(Y |X) is a conditional probability and is verbalized as 
“the probability of Y given X”, whereas the quantity p(X) is a marginal probability 
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and is simply “the probability of X”. These two simple rules form the basis for all 
of the probabilistic machinery that we use throughout this book. 

From the product rule, together with the symmetry property p(X, Y) = p(Y, X), 
we immediately obtain the following relationship between conditional probabilities 


P(X|Y)p(¥) 


P(Y|X) = WX) 


(1.12) 
which is called Bayes’ theorem and which plays a central role in pattern recognition 
and machine learning. Using the sum rule, the denominator in Bayes’ theorem can 
be expressed in terms of the quantities appearing in the numerator 


p(X) = So p(XIY)p(Y). (1.13) 
ye 


We can view the denominator in Bayes’ theorem as being the normalization constant 
required to ensure that the sum of the conditional probability on the left-hand side of 
(1.12) over all values of Y equals one. 

In Figure 1.11, we show a simple example involving a joint distribution over two 
variables to illustrate the concept of marginal and conditional distributions. Here 
a finite sample of N = 60 data points has been drawn from the joint distribution 
and is shown in the top left. In the top right is a histogram of the fractions of data 
points having each of the two values of Y. From the definition of probability, these 
fractions would equal the corresponding probabilities p(Y ) in the limit N — oo. We 
can view the histogram as a simple way to model a probability distribution given only 
a finite number of points drawn from that distribution. Modelling distributions from 
data lies at the heart of statistical pattern recognition and will be explored in great 
detail in this book. The remaining two plots in Figure 1.11 show the corresponding 
histogram estimates of p(X) and p(X|Y = 1). 

Let us now return to our example involving boxes of fruit. For the moment, we 
shall once again be explicit about distinguishing between the random variables and 
their instantiations. We have seen that the probabilities of selecting either the red or 
the blue boxes are given by 


p(B =r) 4/10 (1.14) 
p(B=b) = 6/10 (1.15) 


respectively. Note that these satisfy p(B = r) + p(B = b) = 1. 

Now suppose that we pick a box at random, and it turns out to be the blue box. 
Then the probability of selecting an apple is just the fraction of apples in the blue 
box which is 3/4, and so p(F = a|B = b) = 3/4. In fact, we can write out all four 
conditional probabilities for the type of fruit, given the selected box 


p(F=a|B=r) = 1/4 (1.16) 
p(F=0|B=r) = 3/4 (1.17) 
oF =aB =i = 3/4 (1.18) 
p(F = o|B = b) 1/4. (1.19) 


16 1. INTRODUCTION 


p(X,Y) p(y) 
X 

p(X) p(X|Y = 1) 
x X 


Figure 1.11 An illustration of a distribution over two variables, X, which takes 9 possible values, and Y, which 
takes two possible values. The top left figure shows a sample of 60 points drawn from a joint probability distri- 
bution over these variables. The remaining figures show histogram estimates of the marginal distributions p(X) 
and p(Y), as well as the conditional distribution p(X|Y = 1) corresponding to the bottom row in the top left 
figure. 


Again, note that these probabilities are normalized so that 


p(F = a|B =r)+p(F = oB =r)=1 (1.20) 


and similarly 
p(F = a|B = b)+ p(F = o|B = b) = 1. (1.21) 


We can now use the sum and product rules of probability to evaluate the overall 
probability of choosing an apple 


p(F =a) p(F = a|B = r)p(B = r) + p(F = a|B = b)p(B = b) 
1 4 3 6 U 
= x H x = (1.22) 
4° 104° 10 20 


from which it follows, using the sum rule, that p(F = o) = 1 — 11/20 = 9/20. 
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Suppose instead we are told that a piece of fruit has been selected and it is an 
orange, and we would like to know which box it came from. This requires that 
we evaluate the probability distribution over boxes conditioned on the identity of 
the fruit, whereas the probabilities in (1.16)-(1.19) give the probability distribution 
over the fruit conditioned on the identity of the box. We can solve the problem of 
reversing the conditional probability by using Bayes’ theorem to give 


pF =oBar( Bar) 3.4 20 2 


ie p(F = 0) “aim ar “A 


From the sum rule, it then follows that p(B = b|F = o) = 1 — 2/3 = 1/3. 

We can provide an important interpretation of Bayes’ theorem as follows. If 
we had been asked which box had been chosen before being told the identity of 
the selected item of fruit, then the most complete information we have available is 
provided by the probability p(B). We call this the prior probability because it is the 
probability available before we observe the identity of the fruit. Once we are told that 
the fruit is an orange, we can then use Bayes’ theorem to compute the probability 
p(B|F), which we shall call the posterior probability because it is the probability 
obtained after we have observed F. Note that in this example, the prior probability 
of selecting the red box was 4/10, so that we were more likely to select the blue box 
than the red one. However, once we have observed that the piece of selected fruit is 
an orange, we find that the posterior probability of the red box is now 2/3, so that 
it is now more likely that the box we selected was in fact the red one. This result 
accords with our intuition, as the proportion of oranges is much higher in the red box 
than it is in the blue box, and so the observation that the fruit was an orange provides 
significant evidence favouring the red box. In fact, the evidence is sufficiently strong 
that it outweighs the prior and makes it more likely that the red box was chosen 
rather than the blue one. 

Finally, we note that if the joint distribution of two variables factorizes into the 
product of the marginals, so that p(X,Y) = p(X)p(Y), then X and Y are said to 
be independent. From the product rule, we see that p(Y|X) = p(Y), and so the 
conditional distribution of Y given X is indeed independent of the value of X. For 
instance, in our boxes of fruit example, if each box contained the same fraction of 
apples and oranges, then p(F'|B) = P(F), so that the probability of selecting, say, 
an apple is independent of which box is chosen. 


1.2.1 Probability densities 


As well as considering probabilities defined over discrete sets of events, we 
also wish to consider probabilities with respect to continuous variables. We shall 
limit ourselves to a relatively informal discussion. If the probability of a real-valued 
variable x falling in the interval (x, x + ôx) is given by p(x)dx for 6x — 0, then 
p(x) is called the probability density over x. This is illustrated in Figure 1.12. The 
probability that « will lie in an interval (a, b) is then given by 


b 
p(x € (a, b)) = f p(a) da. (1.24) 
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Figure 1.12 The concept of probability for 
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discrete variables can be ex- 

tended to that of a probability p(z) 
density p(x) over a continuous 

variable x and is such that the 

probability of x lying in the inter- 

val (x, x +ôx) is given by p(x)da 

for ôr — 0. The probability 

density can be expressed as the 

derivative of a cumulative distri- 

bution function P(x). 


ôx T 


Because probabilities are nonnegative, and because the value of x must lie some- 
where on the real axis, the probability density p(x) must satisfy the two conditions 


plz) > 0 (1.25) 
/ plelde = 1. (1.26) 


=00 


Under a nonlinear change of variable, a probability density transforms differently 
from a simple function, due to the Jacobian factor. For instance, if we consider 
a change of variables x = g(y), then a function f(x) becomes f(y) = f(g(y)). 
Now consider a probability density p,(«) that corresponds to a density p,(y) with 
respect to the new variable y, where the suffices denote the fact that p,(a) and p(y) 
are different densities. Observations falling in the range (x, x + ôx) will, for small 
values of ôx, be transformed into the range (y, y + dy) where p,(x)dx ~ py(y)dy, 
and hence 


Py(y) = Pala) 


= px(g(y)) |g (y). (1.27) 


One consequence of this property is that the concept of the maximum of a probability 
density is dependent on the choice of variable. 

The probability that x lies in the interval (—oo, z) is given by the cumulative 
distribution function defined by 


P(z)= [ p(x) da (1.28) 


which satisfies P’(a) = p(x), as shown in Figure 1.12. 
If we have several continuous variables x1, ..., £p, denoted collectively by the 
vector x, then we can define a joint probability density p(x) = p(a1,...,@p) such 
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that the probability of x falling in an infinitesimal volume 6x containing the point x 
is given by p(x)dx. This multivariate probability density must satisfy 


p(x) > 0 (1.29) 
fra D (1.30) 


in which the integral is taken over the whole of x space. We can also consider joint 
probability distributions over a combination of discrete and continuous variables. 

Note that if x is a discrete variable, then p(x) is sometimes called a probability 
mass function because it can be regarded as a set of ‘probability masses’ concentrated 
at the allowed values of x. 

The sum and product rules of probability, as well as Bayes’ theorem, apply 
equally to the case of probability densities, or to combinations of discrete and con- 
tinuous variables. For instance, if x and y are two real variables, then the sum and 
product rules take the form 


plx) = frena (1.31) 
p(z, y) = plyļz)p(z). (1.32) 


A formal justification of the sum and product rules for continuous variables (Feller, 
1966) requires a branch of mathematics called measure theory and lies outside the 
scope of this book. Its validity can be seen informally, however, by dividing each 
real variable into intervals of width A and considering the discrete probability dis- 
tribution over these intervals. Taking the limit A — 0 then turns sums into integrals 
and gives the desired result. 


1.2.2 Expectations and covariances 


One of the most important operations involving probabilities is that of finding 
weighted averages of functions. The average value of some function f(x) under a 
probability distribution p(x) is called the expectation of f(x) and will be denoted by 
E[f]. For a discrete distribution, it is given by 


ELA] = X pla) f(a) (1.33) 


so that the average is weighted by the relative probabilities of the different values 
of x. In the case of continuous variables, expectations are expressed in terms of an 
integration with respect to the corresponding probability density 


ELF] = i pla) f(a) de. (134) 


In either case, if we are given a finite number N of points drawn from the probability 
distribution or probability density, then the expectation can be approximated as a 
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Exercise 1.6 


finite sum over these points 


a LS 
Elf] ~ ap DL flan): (1.35) 


n=1 


We shall make extensive use of this result when we discuss sampling methods in 
Chapter 11. The approximation in (1.35) becomes exact in the limit N — oo. 

Sometimes we will be considering expectations of functions of several variables, 
in which case we can use a subscript to indicate which variable is being averaged 
over, so that for instance 


El f(x, y)] (1.36) 


denotes the average of the function f(z, y) with respect to the distribution of x. Note 
that E,[ f(x, y)] will be a function of y. 

We can also consider a conditional expectation with respect to a conditional 
distribution, so that 


Eelflyl = X v(aly) f(x) (1.37) 


with an analogous definition for continuous variables. 
The variance of f(x) is defined by 


var[f] = E [(f (x) — E[f(2)])"] (1.38) 


and provides a measure of how much variability there is in f(x) around its mean 
value E| f(x)]. Expanding out the square, we see that the variance can also be written 
in terms of the expectations of f(a) and f(x)? 


var[f] = E[f(x)’] — E[f(«))’. (1.39) 


In particular, we can consider the variance of the variable x itself, which is given by 


var|z] = E[z?] — E[z]?. (1.40) 


For two random variables x and y, the covariance is defined by 


cov[x,y] = Ex,y[{# —Ela]} {y — Ely]}] 
= Ex ley] — Ela]E[y] (1.41) 
which expresses the extent to which x and y vary together. If x and y are indepen- 


dent, then their covariance vanishes. 
In the case of two vectors of random variables x and y, the covariance is a matrix 


Exy {x — Ek] Hy" — Ely*]}] 
= E,yl[xy"]—E[x]Ely’]. (1.42) 


cov[x, y] 


If we consider the covariance of the components of a vector x with each other, then 
we use a slightly simpler notation cov[x] = cov[x, x]. 
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1.2.3 Bayesian probabilities 


So far in this chapter, we have viewed probabilities in terms of the frequencies 
of random, repeatable events. We shall refer to this as the classical or frequentist 
interpretation of probability. Now we turn to the more general Bayesian view, in 
which probabilities provide a quantification of uncertainty. 

Consider an uncertain event, for example whether the moon was once in its own 
orbit around the sun, or whether the Arctic ice cap will have disappeared by the end 
of the century. These are not events that can be repeated numerous times in order 
to define a notion of probability as we did earlier in the context of boxes of fruit. 
Nevertheless, we will generally have some idea, for example, of how quickly we 
think the polar ice is melting. If we now obtain fresh evidence, for instance from a 
new Earth observation satellite gathering novel forms of diagnostic information, we 
may revise our opinion on the rate of ice loss. Our assessment of such matters will 
affect the actions we take, for instance the extent to which we endeavour to reduce 
the emission of greenhouse gasses. In such circumstances, we would like to be able 
to quantify our expression of uncertainty and make precise revisions of uncertainty in 
the light of new evidence, as well as subsequently to be able to take optimal actions 
or decisions as a consequence. This can all be achieved through the elegant, and very 
general, Bayesian interpretation of probability. 

The use of probability to represent uncertainty, however, is not an ad-hoc choice, 
but is inevitable if we are to respect common sense while making rational coherent 
inferences. For instance, Cox (1946) showed that if numerical values are used to 
represent degrees of belief, then a simple set of axioms encoding common sense 
properties of such beliefs leads uniquely to a set of rules for manipulating degrees of 
belief that are equivalent to the sum and product rules of probability. This provided 
the first rigorous proof that probability theory could be regarded as an extension of 
Boolean logic to situations involving uncertainty (Jaynes, 2003). Numerous other 
authors have proposed different sets of properties or axioms that such measures of 
uncertainty should satisfy (Ramsey, 1931; Good, 1950; Savage, 1961; deFinetti, 
1970; Lindley, 1982). In each case, the resulting numerical quantities behave pre- 
cisely according to the rules of probability. It is therefore natural to refer to these 
quantities as (Bayesian) probabilities. 

In the field of pattern recognition, too, it is helpful to have a more general no- 


Thomas Bayes 
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tion of probability. Consider the example of polynomial curve fitting discussed in 
Section 1.1. It seems reasonable to apply the frequentist notion of probability to the 
random values of the observed variables t,,. However, we would like to address and 
quantify the uncertainty that surrounds the appropriate choice for the model param- 
eters w. We shall see that, from a Bayesian perspective, we can use the machinery 
of probability theory to describe the uncertainty in model parameters such as w, or 
indeed in the choice of model itself. 

Bayes’ theorem now acquires a new significance. Recall that in the boxes of fruit 
example, the observation of the identity of the fruit provided relevant information 
that altered the probability that the chosen box was the red one. In that example, 
Bayes’ theorem was used to convert a prior probability into a posterior probability 
by incorporating the evidence provided by the observed data. As we shall see in 
detail later, we can adopt a similar approach when making inferences about quantities 
such as the parameters w in the polynomial curve fitting example. We capture our 
assumptions about w, before observing the data, in the form of a prior probability 
distribution p(w). The effect of the observed data D = {t1,..., ty} is expressed 
through the conditional probability p(D|w), and we shall see later, in Section 1.2.5, 
how this can be represented explicitly. Bayes’ theorem, which takes the form 

p(D|w)p(w) 
p(w|D) DD) (1.43) 
then allows us to evaluate the uncertainty in w after we have observed D in the form 
of the posterior probability p(w|D). 

The quantity p(D|w) on the right-hand side of Bayes’ theorem is evaluated for 
the observed data set D and can be viewed as a function of the parameter vector 
w, in which case it is called the likelihood function. It expresses how probable the 
observed data set is for different settings of the parameter vector w. Note that the 
likelihood is not a probability distribution over w, and its integral with respect to w 
does not (necessarily) equal one. 

Given this definition of likelihood, we can state Bayes’ theorem in words 


posterior x likelihood x prior (1.44) 


where all of these quantities are viewed as functions of w. The denominator in 
(1.43) is the normalization constant, which ensures that the posterior distribution 
on the left-hand side is a valid probability density and integrates to one. Indeed, 
integrating both sides of (1.43) with respect to w, we can express the denominator 
in Bayes’ theorem in terms of the prior distribution and the likelihood function 


pD) = i p(Diw)p(w) dw. (1.45) 


In both the Bayesian and frequentist paradigms, the likelihood function p(D|w) 
plays a central role. However, the manner in which it is used is fundamentally dif- 
ferent in the two approaches. In a frequentist setting, w is considered to be a fixed 
parameter, whose value is determined by some form of ‘estimator’, and error bars 


Section 2.1 
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Section 1.3 
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on this estimate are obtained by considering the distribution of possible data sets D. 
By contrast, from the Bayesian viewpoint there is only a single data set D (namely 
the one that is actually observed), and the uncertainty in the parameters is expressed 
through a probability distribution over w. 

A widely used frequentist estimator is maximum likelihood, in which w is set 
to the value that maximizes the likelihood function p(D|w). This corresponds to 
choosing the value of w for which the probability of the observed data set is maxi- 
mized. In the machine learning literature, the negative log of the likelihood function 
is called an error function. Because the negative logarithm is a monotonically de- 
creasing function, maximizing the likelihood is equivalent to minimizing the error. 

One approach to determining frequentist error bars is the bootstrap (Efron, 1979; 
Hastie et al., 2001), in which multiple data sets are created as follows. Suppose our 
original data set consists of N data points X = {x,...,x~}. We can create a new 
data set Xp by drawing N points at random from X, with replacement, so that some 
points in X may be replicated in Xp, whereas other points in X may be absent from 
Xp. This process can be repeated L times to generate L data sets each of size N and 
each obtained by sampling from the original data set X. The statistical accuracy of 
parameter estimates can then be evaluated by looking at the variability of predictions 
between the different bootstrap data sets. 

One advantage of the Bayesian viewpoint is that the inclusion of prior knowl- 
edge arises naturally. Suppose, for instance, that a fair-looking coin is tossed three 
times and lands heads each time. A classical maximum likelihood estimate of the 
probability of landing heads would give 1, implying that all future tosses will land 
heads! By contrast, a Bayesian approach with any reasonable prior will lead to a 
much less extreme conclusion. 

There has been much controversy and debate associated with the relative mer- 
its of the frequentist and Bayesian paradigms, which have not been helped by the 
fact that there is no unique frequentist, or even Bayesian, viewpoint. For instance, 
one common criticism of the Bayesian approach is that the prior distribution is of- 
ten selected on the basis of mathematical convenience rather than as a reflection of 
any prior beliefs. Even the subjective nature of the conclusions through their de- 
pendence on the choice of prior is seen by some as a source of difficulty. Reducing 
the dependence on the prior is one motivation for so-called noninformative priors. 
However, these lead to difficulties when comparing different models, and indeed 
Bayesian methods based on poor choices of prior can give poor results with high 
confidence. Frequentist evaluation methods offer some protection from such prob- 
lems, and techniques such as cross-validation remain useful in areas such as model 
comparison. 

This book places a strong emphasis on the Bayesian viewpoint, reflecting the 
huge growth in the practical importance of Bayesian methods in the past few years, 
while also discussing useful frequentist concepts as required. 

Although the Bayesian framework has its origins in the 18‘ century, the prac- 
tical application of Bayesian methods was for a long time severely limited by the 
difficulties in carrying through the full Bayesian procedure, particularly the need to 
marginalize (sum or integrate) over the whole of parameter space, which, as we shall 
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see, is required in order to make predictions or to compare different models. The 
development of sampling methods, such as Markov chain Monte Carlo (discussed in 
Chapter 11) along with dramatic improvements in the speed and memory capacity 
of computers, opened the door to the practical use of Bayesian techniques in an im- 
pressive range of problem domains. Monte Carlo methods are very flexible and can 
be applied to a wide range of models. However, they are computationally intensive 
and have mainly been used for small-scale problems. 

More recently, highly efficient deterministic approximation schemes such as 
variational Bayes and expectation propagation (discussed in Chapter 10) have been 
developed. These offer a complementary alternative to sampling methods and have 
allowed Bayesian techniques to be used in large-scale applications (Blei et al., 2003). 


1.2.4 The Gaussian distribution 


We shall devote the whole of Chapter 2 to a study of various probability dis- 
tributions and their key properties. It is convenient, however, to introduce here one 
of the most important probability distributions for continuous variables, called the 
normal or Gaussian distribution. We shall make extensive use of this distribution in 
the remainder of this chapter and indeed throughout much of the book. 

For the case of a single real-valued variable x, the Gaussian distribution is de- 
fined by 


1 1 
N (x|, 0°) = CE ex 553 wr (1.46) 


which is governed by two parameters: ju, called the mean, and o°, called the vari- 
ance. The square root of the variance, given by a, is called the standard deviation, 
and the reciprocal of the variance, written as 3 = 1/0, is called the precision. We 
shall see the motivation for these terms shortly. Figure 1.13 shows a plot of the 
Gaussian distribution. 

From the form of (1.46) we see that the Gaussian distribution satisfies 


N(a|u,07) > 0. (1.47) 


Also it is straightforward to show that the Gaussian is normalized, so that 
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Figure 1.13 Plot of the univariate Gaussian 
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showing the mean „u and the | N(z|p,07) 
standard deviation o. 


f N (z|u, 0°) dz =1. (1.48) 


— 00 


Thus (1.46) satisfies the two requirements for a valid probability density. 
We can readily find expectations of functions of x under the Gaussian distribu- 
tion. In particular, the average value of x is given by 


Elz] = f N (x|u, 0°) oda = p. (1.49) 


CO 


Because the parameter u represents the average value of x under the distribution, it 
is referred to as the mean. Similarly, for the second order moment 


E[x?] = f N (a\|p,07) 2? da = p? + 0°. (1.50) 


=00 


From (1.49) and (1.50), it follows that the variance of x is given by 


var[z] = E[x?] — Efx]? = o° (1.51) 


and hence g? is referred to as the variance parameter. The maximum of a distribution 
is known as its mode. For a Gaussian, the mode coincides with the mean. 

We are also interested in the Gaussian distribution defined over a D-dimensional 
vector x of continuous variables, which is given by 


1 1 1 = 
N (x|u, £) = (27) ?/2 |S]1/2 ex { 5 (x > ae E m) (1.52) 


where the D-dimensional vector p is called the mean, the D x D matrix © is called 
the covariance, and |%| denotes the determinant of X. We shall make use of the 
multivariate Gaussian distribution briefly in this chapter, although its properties will 
be studied in detail in Section 2.3. 
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Ln x 


Now suppose that we have a data set of observations X = (a1,...,2y)*, rep- 
resenting N observations of the scalar variable x. Note that we are using the type- 
face X to distinguish this from a single observation of the vector-valued variable 
(£1,.-., £p)", which we denote by x. We shall suppose that the observations are 
drawn independently from a Gaussian distribution whose mean u and variance o? 
are unknown, and we would like to determine these parameters from the data set. 
Data points that are drawn independently from the same distribution are said to be 
independent and identically distributed, which is often abbreviated to i.i.d. We have 
seen that the joint probability of two independent events is given by the product of 
the marginal probabilities for each event separately. Because our data set X is i.i.d., 
we can therefore write the probability of the data set, given u and o°, in the form 


pxu, 07) - [Iw alte) (1.53) 


When viewed as a function of u and o?, this is the likelihood function for the Gaus- 
sian and is interpreted diagrammatically in Figure 1.14. 

One common criterion for determining the parameters in a probability distribu- 
tion using an observed data set is to find the parameter values that maximize the 
likelihood function. This might seem like a strange criterion because, from our fore- 
going discussion of probability theory, it would seem more natural to maximize the 
probability of the parameters given the data, not the probability of the data given the 
parameters. In fact, these two criteria are related, as we shall discuss in the context 
of curve fitting. 

For the moment, however, we shall determine values for the unknown parame- 
ters u and o? in the Gaussian by maximizing the likelihood function (1.53). In prac- 
tice, it is more convenient to maximize the log of the likelihood function. Because 
the logarithm is a monotonically increasing function of its argument, maximization 
of the log of a function is equivalent to maximization of the function itself. Taking 
the log not only simplifies the subsequent mathematical analysis, but it also helps 
numerically because the product of a large number of small probabilities can easily 
underflow the numerical precision of the computer, and this is resolved by computing 
instead the sum of the log probabilities. From (1.46) and (1.53), the log likelihood 
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function can be written in the form 


N 
1 N N 
Inp (xla, 07) oe J (an — u) 5 Ino? 5 In(27). (1.54) 
n=1 


Maximizing (1.54) with respect to u, we obtain the maximum likelihood solution 
given by 


1 
HM = y 3 En (1.55) 


which is the sample mean, i.e., the mean of the observed values {x,,}. Similarly, 
maximizing (1.54) with respect to o°, we obtain the maximum likelihood solution 
for the variance in the form 


Ther, = N DO = umt)? (1.56) 


which is the sample variance measured with respect to the sample mean umr. Note 
that we are performing a joint maximization of (1.54) with respect to u and o°, but 
in the case of the Gaussian distribution the solution for u decouples from that for o? 
so that we can first evaluate (1.55) and then subsequently use this result to evaluate 
(1.56). 

Later in this chapter, and also in subsequent chapters, we shall highlight the sig- 
nificant limitations of the maximum likelihood approach. Here we give an indication 
of the problem in the context of our solutions for the maximum likelihood param- 
eter settings for the univariate Gaussian distribution. In particular, we shall show 
that the maximum likelihood approach systematically underestimates the variance 
of the distribution. This is an example of a phenomenon called bias and is related 
to the problem of over-fitting encountered in the context of polynomial curve fitting. 
We first note that the maximum likelihood solutions um and o%,;, are functions of 
the data set values z1,..., £y. Consider the expectations of these quantities with 
respect to the data set values, which themselves come from a Gaussian distribution 
with parameters u and o°. It is straightforward to show that 


Elumi] = y (1.57) 
Nea 


Elou] = (=F) o?’ (1.58) 


so that on average the maximum likelihood estimate will obtain the correct mean but 
will underestimate the true variance by a factor (N — 1)/N. The intuition behind 
this result is given by Figure 1.15. 

From (1.58) it follows that the following estimate for the variance parameter is 
unbiased 


N 


1 
=I] N (2n — pm)’. (1.59) 


n=1 


z N 
a = Woe 
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Illustration of how bias arises in using max- 

imum likelihood to determine the variance 

of a Gaussian. The green curve shows 

the true Gaussian distribution from which 

data is generated, and the three red curves 

show the Gaussian distributions obtained (a) 
by fitting to three data sets, each consist- 

ing of two data points shown in blue, us- 

ing the maximum likelihood results (1.55) 

and (1.56). Averaged across the three data 

sets, the mean is correct, but the variance 

is systematically under-estimated because (b) 
it is measured relative to the sample mean 

and not relative to the true mean. 


(c) 


In Section 10.1.3, we shall see how this result arises automatically when we adopt a 
Bayesian approach. 

Note that the bias of the maximum likelihood solution becomes less significant 
as the number NV of data points increases, and in the limit NV — oo the maximum 
likelihood solution for the variance equals the true variance of the distribution that 
generated the data. In practice, for anything other than small N, this bias will not 
prove to be a serious problem. However, throughout this book we shall be interested 
in more complex models with many parameters, for which the bias problems asso- 
ciated with maximum likelihood will be much more severe. In fact, as we shall see, 
the issue of bias in maximum likelihood lies at the root of the over-fitting problem 
that we encountered earlier in the context of polynomial curve fitting. 


1.2.5 Curve fitting re-visited 


We have seen how the problem of polynomial curve fitting can be expressed in 
terms of error minimization. Here we return to the curve fitting example and view it 
from a probabilistic perspective, thereby gaining some insights into error functions 
and regularization, as well as taking us towards a full Bayesian treatment. 

The goal in the curve fitting problem is to be able to make predictions for the 
target variable t given some new value of the input variable x on the basis of a set of 
training data comprising N input values X = (x1, . . . , æy)” and their corresponding 
target values t = (t1,..., tn)". We can express our uncertainty over the value of 
the target variable using a probability distribution. For this purpose, we shall assume 
that, given the value of x, the corresponding value of ¢ has a Gaussian distribution 
with a mean equal to the value y(x, w) of the polynomial curve given by (1.1). Thus 
we have 


p(t|z, w, 8) =N (tly(x, w), *) (1.60) 


where, for consistency with the notation in later chapters, we have defined a preci- 
sion parameter 6 corresponding to the inverse variance of the distribution. This is 
illustrated schematically in Figure 1.16. 
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Figure 1.16 Schematic illustration of a Gaus- 
sian conditional distribution for t given x given by 
(1.60), in which the mean is given by the polyno- 
mial function y(x, w), and the precision is given 
by the parameter 3, which is related to the vari- 
ance by 87t! = o? 


y(zo, w 


p(t|zo, w, 8) 


XO x 


We now use the training data {x,t} to determine the values of the unknown 
parameters w and 3 by maximum likelihood. If the data are assumed to be drawn 
independently from the distribution (1.60), then the likelihood function is given by 


p(t|x, w, 8) = Ivi (trly(tn, w), B+). (1.61) 


n=1 


As we did in the case of the simple Gaussian distribution earlier, it is convenient to 
maximize the logarithm of the likelihood function. Substituting for the form of the 
Gaussian distribution, given by (1.46), we obtain the log likelihood function in the 
form 


N 
In p(t|x, w, 3) = “7 y(&n,w) — tn}? 4 A ng A n(2n), (1.62) 


Consider first the determination of the maximum likelihood solution for the polyno- 
mial coefficients, which will be denoted by wm. These are determined by maxi- 
mizing (1.62) with respect to w. For this purpose, we can omit the last two terms 
on the right-hand side of (1.62) because they do not depend on w. Also, we note 
that scaling the log likelihood by a positive constant coefficient does not alter the 
location of the maximum with respect to w, and so we can replace the coefficient 
6/2 with 1/2. Finally, instead of maximizing the log likelihood, we can equivalently 
minimize the negative log likelihood. We therefore see that maximizing likelihood is 
equivalent, so far as determining w is concerned, to minimizing the sum-of-squares 
error function defined by (1.2). Thus the sum-of-squares error function has arisen as 
a consequence of maximizing likelihood under the assumption of a Gaussian noise 
distribution. 

We can also use maximum likelihood to determine the precision parameter ( of 
the Gaussian conditional distribution. Maximizing (1.62) with respect to ( gives 


1 


N 
il 
Bar, =J X {y(n WML) — tn¥ (1.63) 
n=1 
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Again we can first determine the parameter vector wmz, governing the mean and sub- 
sequently use this to find the precision mı as was the case for the simple Gaussian 
distribution. 

Having determined the parameters w and 8, we can now make predictions for 
new values of x. Because we now have a probabilistic model, these are expressed 
in terms of the predictive distribution that gives the probability distribution over t, 
rather than simply a point estimate, and is obtained by substituting the maximum 
likelihood parameters into (1.60) to give 


p(tlz, Wmr, Gu) = N (tly(@, wmr), Buz) - (1.64) 


Now let us take a step towards a more Bayesian approach and introduce a prior 
distribution over the polynomial coefficients w. For simplicity, let us consider a 
Gaussian distribution of the form 


M+1)/2 
p(wl|a) = N(w|0,07'T) = C k exp {-Sw'w} (1.65) 
20 2 
where a is the precision of the distribution, and M +1 is the total number of elements 
in the vector w for an M*‘ order polynomial. Variables such as a, which control 
the distribution of model parameters, are called hyperparameters. Using Bayes’ 
theorem, the posterior distribution for w is proportional to the product of the prior 
distribution and the likelihood function 


P(w]X, t, a, 3) x p(t|x, w, 8)p(wla). (1.66) 


We can now determine w by finding the most probable value of w given the data, 
in other words by maximizing the posterior distribution. This technique is called 
maximum posterior, or simply MAP. Taking the negative logarithm of (1.66) and 
combining with (1.62) and (1.65), we find that the maximum of the posterior is 
given by the minimum of 


N 
5 Lil w) — tn} + Sw'w. (1.67) 


Thus we see that maximizing the posterior distribution is equivalent to minimizing 
the regularized sum-of-squares error function encountered earlier in the form (1.4), 
with a regularization parameter given by A = a/ 2. 


1.2.6 Bayesian curve fitting 


Although we have included a prior distribution p(w|a), we are so far still mak- 
ing a point estimate of w and so this does not yet amount to a Bayesian treatment. In 
a fully Bayesian approach, we should consistently apply the sum and product rules 
of probability, which requires, as we shall see shortly, that we integrate over all val- 
ues of w. Such marginalizations lie at the heart of Bayesian methods for pattern 
recognition. 
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In the curve fitting problem, we are given the training data x and t, along with 
a new test point x, and our goal is to predict the value of t. We therefore wish 
to evaluate the predictive distribution p(t|x,x,t). Here we shall assume that the 
parameters a and ĝ are fixed and known in advance (in later chapters we shall discuss 
how such parameters can be inferred from data in a Bayesian setting). 

A Bayesian treatment simply corresponds to a consistent application of the sum 
and product rules of probability, which allow the predictive distribution to be written 
in the form 


p(t|a,x,t) = J vite. wewn dw. (1.68) 


Here p(t|x, w) is given by (1.60), and we have omitted the dependence on a and 
B to simplify the notation. Here p(w|x,t) is the posterior distribution over param- 
eters, and can be found by normalizing the right-hand side of (1.66). We shall see 
in Section 3.3 that, for problems such as the curve-fitting example, this posterior 
distribution is a Gaussian and can be evaluated analytically. Similarly, the integra- 
tion in (1.68) can also be performed analytically with the result that the predictive 
distribution is given by a Gaussian of the form 


p(t|ax, x,t) =N (t|m(z), ° (x)) (1.69) 


where the mean and variance are given by 


N 
mx) = Bol) SY P(an)tn (1.70) 
n=1 
s(x) = B + (z)'Sd(z2). (1.71) 
Here the matrix S is given by 

N 
S~ =al+ BS) Pan) d(x)" (1.72) 

n=1 


where I is the unit matrix, and we have defined the vector @(x) with elements 
il£) = zt for i = 0,..., M. 

We see that the variance, as well as the mean, of the predictive distribution in 
(1.69) is dependent on x. The first term in (1.71) represents the uncertainty in the 
predicted value of t due to the noise on the target variables and was expressed already 
in the maximum likelihood predictive distribution (1.64) through 3,;;,. However, the 
second term arises from the uncertainty in the parameters w and is a consequence 
of the Bayesian treatment. The predictive distribution for the synthetic sinusoidal 
regression problem is illustrated in Figure 1.17. 


32 


1. INTRODUCTION 


Figure 1.17 The predictive distribution result- 


1.3. 


ing from a Bayesian treatment of 
polynomial curve fitting using an 
M = 9 polynomial, with the fixed 
parametersa=5x10%andB= ¢ 
11.1 (corresponding to the known 
noise variance), in which the red O\\O 
curve denotes the mean of the 9 
predictive distribution and the red 
region corresponds to +1 stan- 
dard deviation around the mean. 


—1+ 


Model Selection 


In our example of polynomial curve fitting using least squares, we saw that there was 
an optimal order of polynomial that gave the best generalization. The order of the 
polynomial controls the number of free parameters in the model and thereby governs 
the model complexity. With regularized least squares, the regularization coefficient 
A also controls the effective complexity of the model, whereas for more complex 
models, such as mixture distributions or neural networks there may be multiple pa- 
rameters governing complexity. In a practical application, we need to determine 
the values of such parameters, and the principal objective in doing so is usually to 
achieve the best predictive performance on new data. Furthermore, as well as find- 
ing the appropriate values for complexity parameters within a given model, we may 
wish to consider a range of different types of model in order to find the best one for 
our particular application. 

We have already seen that, in the maximum likelihood approach, the perfor- 
mance on the training set is not a good indicator of predictive performance on un- 
seen data due to the problem of over-fitting. If data is plentiful, then one approach is 
simply to use some of the available data to train a range of models, or a given model 
with a range of values for its complexity parameters, and then to compare them on 
independent data, sometimes called a validation set, and select the one having the 
best predictive performance. If the model design is iterated many times using a lim- 
ited size data set, then some over-fitting to the validation data can occur and so it may 
be necessary to keep aside a third test set on which the performance of the selected 
model is finally evaluated. 

In many applications, however, the supply of data for training and testing will be 
limited, and in order to build good models, we wish to use as much of the available 
data as possible for training. However, if the validation set is small, it will give a 
relatively noisy estimate of predictive performance. One solution to this dilemma is 
to use cross-validation, which is illustrated in Figure 1.18. This allows a proportion 
(S — 1)/S of the available data to be used for training while making use of all of the 
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Figure 1.18 The technique of S-fold cross-validation, illus- | | |] )] run 1 
trated here for the case of S = 4, involves tak- 


1.4. 


ing the available data and partitioning it into S 
groups (in the simplest case these are of equal OC [oy |] )] run 2 
size). Then S — 1 of the groups are used to train 


a set of models that are then evaluated on the re- [ | P |) run 3 


maining group. This procedure is then repeated 


for all S possible choices for the held-out group, CI |] P run 4 


indicated here by the red blocks, and the perfor- 
mance scores from the S runs are then averaged. 


data to assess performance. When data is particularly scarce, it may be appropriate 
to consider the case S = N, where N is the total number of data points, which gives 
the leave-one-out technique. 

One major drawback of cross-validation is that the number of training runs that 
must be performed is increased by a factor of S, and this can prove problematic for 
models in which the training is itself computationally expensive. A further problem 
with techniques such as cross-validation that use separate data to assess performance 
is that we might have multiple complexity parameters for a single model (for in- 
stance, there might be several regularization parameters). Exploring combinations 
of settings for such parameters could, in the worst case, require a number of training 
runs that is exponential in the number of parameters. Clearly, we need a better ap- 
proach. Ideally, this should rely only on the training data and should allow multiple 
hyperparameters and model types to be compared in a single training run. We there- 
fore need to find a measure of performance which depends only on the training data 
and which does not suffer from bias due to over-fitting. 

Historically various ‘information criteria’ have been proposed that attempt to 
correct for the bias of maximum likelihood by the addition of a penalty term to 
compensate for the over-fitting of more complex models. For example, the Akaike 
information criterion, or AIC (Akaike, 1974), chooses the model for which the quan- 
tity 

In p(D|wmt) — M (1.73) 
is largest. Here p(D|wyz) is the best-fit log likelihood, and M is the number of 
adjustable parameters in the model. A variant of this quantity, called the Bayesian 
information criterion, or BIC, will be discussed in Section 4.4.1. Such criteria do 
not take account of the uncertainty in the model parameters, however, and in practice 
they tend to favour overly simple models. We therefore turn in Section 3.4 to a fully 
Bayesian approach where we shall see how complexity penalties arise in a natural 
and principled way. 


The Curse of Dimensionality 


In the polynomial curve fitting example we had just one input variable x. For prac- 
tical applications of pattern recognition, however, we will have to deal with spaces 
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Figure 1.19 Scatter plot of the oil flow data 
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for input variables xg and x7, in 
which red denotes the ‘homoge- 
nous’ class, green denotes the 
‘annular class, and blue denotes 
the ‘laminar’ class. Our goal is 
to classify the new test point de- 
noted by ‘x’. 


of high dimensionality comprising many input variables. As we now discuss, this 
poses some serious challenges and is an important factor influencing the design of 
pattern recognition techniques. 

In order to illustrate the problem we consider a synthetically generated data set 
representing measurements taken from a pipeline containing a mixture of oil, wa- 
ter, and gas (Bishop and James, 1993). These three materials can be present in one 
of three different geometrical configurations known as ‘homogenous’, ‘annular’, and 
‘laminar’, and the fractions of the three materials can also vary. Each data point com- 
prises a 12-dimensional input vector consisting of measurements taken with gamma 
ray densitometers that measure the attenuation of gamma rays passing along nar- 
row beams through the pipe. This data set is described in detail in Appendix A. 
Figure 1.19 shows 100 points from this data set on a plot showing two of the mea- 
surements xg and 27 (the remaining ten input values are ignored for the purposes of 
this illustration). Each data point is labelled according to which of the three geomet- 
rical classes it belongs to, and our goal is to use this data as a training set in order to 
be able to classify a new observation (£6, 77), such as the one denoted by the cross 
in Figure 1.19. We observe that the cross is surrounded by numerous red points, and 
so we might suppose that it belongs to the red class. However, there are also plenty 
of green points nearby, so we might think that it could instead belong to the green 
class. It seems unlikely that it belongs to the blue class. The intuition here is that the 
identity of the cross should be determined more strongly by nearby points from the 
training set and less strongly by more distant points. In fact, this intuition turns out 
to be reasonable and will be discussed more fully in later chapters. 

How can we turn this intuition into a learning algorithm? One very simple ap- 
proach would be to divide the input space into regular cells, as indicated in Fig- 
ure 1.20. When we are given a test point and we wish to predict its class, we first 
decide which cell it belongs to, and we then find all of the training data points that 
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Figure 1.21 Illustration of the 
curse of dimensionality, showing 
how the number of regions of a 
regular grid grows exponentially 
with the dimensionality D of the 
space. For clarity, only a subset of 
the cubical regions are shown for 


D=3. 
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Illustration of a simple approach 
to the solution of a classification 
problem in which the input space 
is divided into cells and any new 
test point is assigned to the class 
that has a majority number of rep- 
resentatives in the same cell as 
the test point. As we shall see 
shortly, this simplistic approach 
has some severe shortcomings. 


fall in the same cell. The identity of the test point is predicted as being the same 
as the class having the largest number of training points in the same cell as the test 
point (with ties being broken at random). 

There are numerous problems with this naive approach, but one of the most se- 
vere becomes apparent when we consider its extension to problems having larger 
numbers of input variables, corresponding to input spaces of higher dimensionality. 
The origin of the problem is illustrated in Figure 1.21, which shows that, if we divide 
a region of a space into regular cells, then the number of such cells grows exponen- 
tially with the dimensionality of the space. The problem with an exponentially large 
number of cells is that we would need an exponentially large quantity of training data 
in order to ensure that the cells are not empty. Clearly, we have no hope of applying 
such a technique in a space of more than a few variables, and so we need to find a 
more sophisticated approach. 

We can gain further insight into the problems of high-dimensional spaces by 
returning to the example of polynomial curve fitting and considering how we would 
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extend this approach to deal with input spaces having several variables. If we have 
D input variables, then a general polynomial with coefficients up to order 3 would 
take the form 


D D OD 


y(x, w) =m + oun +O we Zj + 22” WijkLiljEk. (1.74) 
9=1 k=1 


i=1 j=l i=1 


As D increases, so the number of independent coefficients (not all of the coefficients 
are independent due to interchange symmetries amongst the x variables) grows pro- 
portionally to D°. In practice, to capture complex dependencies in the data, we may 
need to use a higher-order polynomial. For a polynomial of order M, the growth in 
the number of coefficients is like DY. Although this is now a power law growth, 
rather than an exponential growth, it still points to the method becoming rapidly 
unwieldy and of limited practical utility. 

Our geometrical intuitions, formed through a life spent in a space of three di- 
mensions, can fail badly when we consider spaces of higher dimensionality. As a 
simple example, consider a sphere of radius r = 1 in a space of D dimensions, and 
ask what is the fraction of the volume of the sphere that lies between radius r = 1 — € 
and r = 1. We can evaluate this fraction by noting that the volume of a sphere of 
radius r in D dimensions must scale as r?, and so we write 


Vp(r) = Kpr” (1.75) 
where the constant Kp depends only on D. Thus the required fraction is given by 


Vp(1) = Vp(1 = €) 
Vp(1) 


which is plotted as a function of e for various values of D in Figure 1.22. We see 
that, for large D, this fraction tends to 1 even for small values of e. Thus, in spaces 
of high dimensionality, most of the volume of a sphere is concentrated in a thin shell 
near the surface! 

As a further example, of direct relevance to pattern recognition, consider the 
behaviour of a Gaussian distribution in a high-dimensional space. If we transform 
from Cartesian to polar coordinates, and then integrate out the directional variables, 
we obtain an expression for the density p(r) as a function of radius r from the origin. 
Thus p(r)dr is the probability mass inside a thin shell of thickness dr located at 
radius r. This distribution is plotted, for various values of D, in Figure 1.23, and we 
see that for large D the probability mass of the Gaussian is concentrated in a thin 
shell. 

The severe difficulty that can arise in spaces of many dimensions is sometimes 
called the curse of dimensionality (Bellman, 1961). In this book, we shall make ex- 
tensive use of illustrative examples involving input spaces of one or two dimensions, 
because this makes it particularly easy to illustrate the techniques graphically. The 
reader should be warned, however, that not all intuitions developed in spaces of low 
dimensionality will generalize to spaces of many dimensions. 


== =<)? (1.76) 


Figure 1.22 Plot of the fraction of the volume of 


Figure 1.23 
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a sphere lying in the range r = 1—« 
to r = 1 for various values of the 
dimensionality D. 


volume fraction 


Although the curse of dimensionality certainly raises important issues for pat- 
tern recognition applications, it does not prevent us from finding effective techniques 
applicable to high-dimensional spaces. The reasons for this are twofold. First, real 
data will often be confined to a region of the space having lower effective dimension- 
ality, and in particular the directions over which important variations in the target 
variables occur may be so confined. Second, real data will typically exhibit some 
smoothness properties (at least locally) so that for the most part small changes in the 
input variables will produce small changes in the target variables, and so we can ex- 
ploit local interpolation-like techniques to allow us to make predictions of the target 
variables for new values of the input variables. Successful pattern recognition tech- 
niques exploit one or both of these properties. Consider, for example, an application 
in manufacturing in which images are captured of identical planar objects on a con- 
veyor belt, in which the goal is to determine their orientation. Each image is a point 


Plot of the probability density with 
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in a high-dimensional space whose dimensionality is determined by the number of 
pixels. Because the objects can occur at different positions within the image and 
in different orientations, there are three degrees of freedom of variability between 
images, and a set of images will live on a three dimensional manifold embedded 
within the high-dimensional space. Due to the complex relationships between the 
object position or orientation and the pixel intensities, this manifold will be highly 
nonlinear. If the goal is to learn a model that can take an input image and output the 
orientation of the object irrespective of its position, then there is only one degree of 
freedom of variability within the manifold that is significant. 


Decision Theory 


We have seen in Section 1.2 how probability theory provides us with a consistent 
mathematical framework for quantifying and manipulating uncertainty. Here we 
turn to a discussion of decision theory that, when combined with probability theory, 
allows us to make optimal decisions in situations involving uncertainty such as those 
encountered in pattern recognition. 

Suppose we have an input vector x together with a corresponding vector t of 
target variables, and our goal is to predict t given a new value for x. For regression 
problems, t will comprise continuous variables, whereas for classification problems 
t will represent class labels. The joint probability distribution p(x,t) provides a 
complete summary of the uncertainty associated with these variables. Determination 
of p(x,t) from a set of training data is an example of inference and is typically a 
very difficult problem whose solution forms the subject of much of this book. In 
a practical application, however, we must often make a specific prediction for the 
value of t, or more generally take a specific action based on our understanding of the 
values t is likely to take, and this aspect is the subject of decision theory. 

Consider, for example, a medical diagnosis problem in which we have taken an 
X-ray image of a patient, and we wish to determine whether the patient has cancer 
or not. In this case, the input vector x is the set of pixel intensities in the image, 
and output variable ¢ will represent the presence of cancer, which we denote by the 
class C4, or the absence of cancer, which we denote by the class C2. We might, for 
instance, choose t to be a binary variable such that t = 0 corresponds to class Cı and 
t = 1 corresponds to class C2. We shall see later that this choice of label values is 
particularly convenient for probabilistic models. The general inference problem then 
involves determining the joint distribution p(x, Cx), or equivalently p(x, t), which 
gives us the most complete probabilistic description of the situation. Although this 
can be a very useful and informative quantity, in the end we must decide either to 
give treatment to the patient or not, and we would like this choice to be optimal 
in some appropriate sense (Duda and Hart, 1973). This is the decision step, and 
it is the subject of decision theory to tell us how to make optimal decisions given 
the appropriate probabilities. We shall see that the decision stage is generally very 
simple, even trivial, once we have solved the inference problem. 

Here we give an introduction to the key ideas of decision theory as required for 
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the rest of the book. Further background, as well as more detailed accounts, can be 
found in Berger (1985) and Bather (2000). 

Before giving a more detailed analysis, let us first consider informally how we 
might expect probabilities to play a role in making decisions. When we obtain the 
X-ray image x for a new patient, our goal is to decide which of the two classes to 
assign to the image. We are interested in the probabilities of the two classes given 
the image, which are given by p(C;|x). Using Bayes’ theorem, these probabilities 
can be expressed in the form 


p(xlCn)p(Ce) 


p(Cxlx) = p(x) 


(1.77) 
Note that any of the quantities appearing in Bayes’ theorem can be obtained from 
the joint distribution p(x, Cp) by either marginalizing or conditioning with respect to 
the appropriate variables. We can now interpret p(C;,) as the prior probability for the 
class Cx, and p(C;,|x) as the corresponding posterior probability. Thus p(C1) repre- 
sents the probability that a person has cancer, before we take the X-ray measurement. 
Similarly, p(C;|x) is the corresponding probability, revised using Bayes’ theorem in 
light of the information contained in the X-ray. If our aim is to minimize the chance 
of assigning x to the wrong class, then intuitively we would choose the class having 
the higher posterior probability. We now show that this intuition is correct, and we 
also discuss more general criteria for making decisions. 


1.5.1 Minimizing the misclassification rate 


Suppose that our goal is simply to make as few misclassifications as possible. 
We need a rule that assigns each value of x to one of the available classes. Such a 
rule will divide the input space into regions Ry called decision regions, one for each 
class, such that all points in Rẹ are assigned to class Cp. The boundaries between 
decision regions are called decision boundaries or decision surfaces. Note that each 
decision region need not be contiguous but could comprise some number of disjoint 
regions. We shall encounter examples of decision boundaries and decision regions in 
later chapters. In order to find the optimal decision rule, consider first of all the case 
of two classes, as in the cancer problem for instance. A mistake occurs when an input 
vector belonging to class C; is assigned to class C or vice versa. The probability of 
this occurring is given by 


p(mistake) = p(x € Ry,C2) + p(x € Re, C1) 
= f p(x, C2) dx + f p(x,Cı) dx. (1.78) 
Ri 


Rə 


We are free to choose the decision rule that assigns each point x to one of the two 
classes. Clearly to minimize p(mistake) we should arrange that each x is assigned to 
whichever class has the smaller value of the integrand in (1.78). Thus, if p(x, C1) > 
p(x, C2) for a given value of x, then we should assign that x to class C4. From the 
product rule of probability we have p(x,C) = p(Cx|x)p(x). Because the factor 
p(x) is common to both terms, we can restate this result as saying that the minimum 
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Figure 1.24 Schematic illustration of the joint probabilities p(2,C,) for each of two classes plotted 


against x, together with the decision boundary x = z. Values of x > @ are classified as 
class C2 and hence belong to decision region R2, whereas points x < Ẹ are classified 
as Cı and belong to Rı. Errors arise from the blue, green, and red regions, so that for 
x < @ the errors are due to points from class C2 being misclassified as Cı (represented by 
the sum of the red and green regions), and conversely for points in the region x > @ the 
errors are due to points from class Cı being misclassified as C2 (represented by the blue 
region). As we vary the location £ of the decision boundary, the combined areas of the 
blue and green regions remains constant, whereas the size of the red region varies. The 
optimal choice for £ is where the curves for p(x,C1) and p(x, C2) cross, corresponding to 
T = xo, because in this case the red region disappears. This is equivalent to the minimum 
misclassification rate decision rule, which assigns each value of x to the class having the 
higher posterior probability p(Cx|z). 


probability of making a mistake is obtained if each value of x is assigned to the class 
for which the posterior probability p(C;,|x) is largest. This result is illustrated for 
two classes, and a single input variable x, in Figure 1.24. 

For the more general case of K classes, it is slightly easier to maximize the 
probability of being correct, which is given by 


K 
p(correct) = Spe E Rk, Cp) 


k=1 
K 

= oa p(x, Cp) dx (1.79) 
k=1 ” Re 


which is maximized when the regions Rẹ are chosen such that each x is assigned 
to the class for which p(x, C) is largest. Again, using the product rule p(x, Cp) = 
p(Ck|x)p(x), and noting that the factor of p(x) is common to all terms, we see 
that each x should be assigned to the class having the largest posterior probability 


P(Cx|x). 
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Figure 1.25 An example of a loss matrix with ele- cancer normal 


ments Lx; for the cancer treatment problem. The rows 
correspond to the true class, whereas the columns cor- cancer 0 1000 
respond to the assignment of class made by our deci- normal 1 0 


sion criterion. 


1.5.2 Minimizing the expected loss 


For many applications, our objective will be more complex than simply mini- 
mizing the number of misclassifications. Let us consider again the medical diagnosis 
problem. We note that, if a patient who does not have cancer is incorrectly diagnosed 
as having cancer, the consequences may be some patient distress plus the need for 
further investigations. Conversely, if a patient with cancer is diagnosed as healthy, 
the result may be premature death due to lack of treatment. Thus the consequences 
of these two types of mistake can be dramatically different. It would clearly be better 
to make fewer mistakes of the second kind, even if this was at the expense of making 
more mistakes of the first kind. 

We can formalize such issues through the introduction of a loss function, also 
called a cost function, which is a single, overall measure of loss incurred in taking 
any of the available decisions or actions. Our goal is then to minimize the total loss 
incurred. Note that some authors consider instead a utility function, whose value 
they aim to maximize. These are equivalent concepts if we take the utility to be 
simply the negative of the loss, and throughout this text we shall use the loss function 
convention. Suppose that, for a new value of x, the true class is Cy and that we assign 
x to class C; (where j may or may not be equal to k). In so doing, we incur some 
level of loss that we denote by L;,;, which we can view as the k, 7 element of a loss 
matrix. For instance, in our cancer example, we might have a loss matrix of the form 
shown in Figure 1.25. This particular loss matrix says that there is no loss incurred 
if the correct decision is made, there is a loss of 1 if a healthy patient is diagnosed as 
having cancer, whereas there is a loss of 1000 if a patient having cancer is diagnosed 
as healthy. 

The optimal solution is the one which minimizes the loss function. However, 
the loss function depends on the true class, which is unknown. For a given input 
vector x, our uncertainty in the true class is expressed through the joint probability 
distribution p(x, Cx) and so we seek instead to minimize the average loss, where the 
average is computed with respect to this distribution, which is given by 


E[L] = yyy Lrjp(x, Cp) ax. (1.80) 
k j J 


Each x can be assigned independently to one of the decision regions Rj. Our goal 
is to choose the regions R; in order to minimize the expected loss (1.80), which 
implies that for each x we should minimize $`, Lyj;p(x, Cp). As before, we can use 
the product rule p(x, Ck) = p(C,|x)p(x) to eliminate the common factor of p(x). 
Thus the decision rule that minimizes the expected loss is the one that assigns each 
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Figure 1.26 


Exercise 1.24 


Illustration of the reject option. Inputs 

x such that the larger of the two poste- 1.0 
rior probabilities is less than or equalto g 
some threshold 0 will be rejected. 


0.0 
reject region 


new x to the class 7 for which the quantity 
S Lrjp(Crlx) (1.81) 
k 


is aminimum. This is clearly trivial to do, once we know the posterior class proba- 
bilities p(C;,|x). 


1.5.3 The reject option 


We have seen that classification errors arise from the regions of input space 
where the largest of the posterior probabilities p(C;,|x) is significantly less than unity, 
or equivalently where the joint distributions p(x, Cp) have comparable values. These 
are the regions where we are relatively uncertain about class membership. In some 
applications, it will be appropriate to avoid making decisions on the difficult cases 
in anticipation of a lower error rate on those examples for which a classification de- 
cision is made. This is known as the reject option. For example, in our hypothetical 
medical illustration, it may be appropriate to use an automatic system to classify 
those X-ray images for which there is little doubt as to the correct class, while leav- 
ing a human expert to classify the more ambiguous cases. We can achieve this by 
introducing a threshold @ and rejecting those inputs x for which the largest of the 
posterior probabilities p(C;,|x) is less than or equal to 6. This is illustrated for the 
case of two classes, and a single continuous input variable x, in Figure 1.26. Note 
that setting 0 = 1 will ensure that all examples are rejected, whereas if there are K 
classes then setting 0 < 1/K will ensure that no examples are rejected. Thus the 
fraction of examples that get rejected is controlled by the value of 6. 

We can easily extend the reject criterion to minimize the expected loss, when 
a loss matrix is given, taking account of the loss incurred when a reject decision is 
made. 


1.5.4 Inference and decision 


We have broken the classification problem down into two separate stages, the 
inference stage in which we use training data to learn a model for p(C;,|x), and the 
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subsequent decision stage in which we use these posterior probabilities to make op- 
timal class assignments. An alternative possibility would be to solve both problems 
together and simply learn a function that maps inputs x directly into decisions. Such 
a function is called a discriminant function. 

In fact, we can identify three distinct approaches to solving decision problems, 
all of which have been used in practical applications. These are given, in decreasing 
order of complexity, by: 


(a) First solve the inference problem of determining the class-conditional densities 
p 
p(x|Ck) for each class Cx individually. Also separately infer the prior class 
probabilities p(C;,). Then use Bayes’ theorem in the form 


P(Cx |x) = nee) 


(1.82) 


to find the posterior class probabilities p(C;,|x). As usual, the denominator 
in Bayes’ theorem can be found in terms of the quantities appearing in the 
numerator, because 


p(x) = X p(x|Cx)p(Cr)- (1.83) 
k 


Equivalently, we can model the joint distribution p(x, Cx) directly and then 
normalize to obtain the posterior probabilities. Having found the posterior 
probabilities, we use decision theory to determine class membership for each 
new input x. Approaches that explicitly or implicitly model the distribution of 
inputs as well as outputs are known as generative models, because by sampling 
from them it is possible to generate synthetic data points in the input space. 


(b) First solve the inference problem of determining the posterior class probabilities 
p(C;,|x), and then subsequently use decision theory to assign each new x to 
one of the classes. Approaches that model the posterior probabilities directly 
are called discriminative models. 


(c) Find a function f(x), called a discriminant function, which maps each input x 
directly onto a class label. For instance, in the case of two-class problems, 
f(-) might be binary valued and such that f = 0 represents class C, and f = 1 
represents class C2. In this case, probabilities play no role. 


Let us consider the relative merits of these three alternatives. Approach (a) is the 
most demanding because it involves finding the joint distribution over both x and 
Ck. For many applications, x will have high dimensionality, and consequently we 
may need a large training set in order to be able to determine the class-conditional 
densities to reasonable accuracy. Note that the class priors p(C;) can often be esti- 
mated simply from the fractions of the training set data points in each of the classes. 
One advantage of approach (a), however, is that it also allows the marginal density 
of data p(x) to be determined from (1.83). This can be useful for detecting new data 
points that have low probability under the model and for which the predictions may 
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Figure 1.27 Example of the class-conditional densities for two classes having a single input variable x (left 
plot) together with the corresponding posterior probabilities (right plot). Note that the left-hand mode of the 
class-conditional density p(x|C1), shown in blue on the left plot, has no effect on the posterior probabilities. The 
vertical green line in the right plot shows the decision boundary in x that gives the minimum misclassification 
rate. 


be of low accuracy, which is known as outlier detection or novelty detection (Bishop, 
1994; Tarassenko, 1995). 

However, if we only wish to make classification decisions, then it can be waste- 
ful of computational resources, and excessively demanding of data, to find the joint 
distribution p(x,C;,) when in fact we only really need the posterior probabilities 
p(Cx|x), which can be obtained directly through approach (b). Indeed, the class- 
conditional densities may contain a lot of structure that has little effect on the pos- 
terior probabilities, as illustrated in Figure 1.27. There has been much interest in 
exploring the relative merits of generative and discriminative approaches to machine 
learning, and in finding ways to combine them (Jebara, 2004; Lasserre et al., 2006). 

An even simpler approach is (c) in which we use the training data to find a 
discriminant function f(x) that maps each x directly onto a class label, thereby 
combining the inference and decision stages into a single learning problem. In the 
example of Figure 1.27, this would correspond to finding the value of x shown by 
the vertical green line, because this is the decision boundary giving the minimum 
probability of misclassification. 

With option (c), however, we no longer have access to the posterior probabilities 
p(Cx|x). There are many powerful reasons for wanting to compute the posterior 
probabilities, even if we subsequently use them to make decisions. These include: 


Minimizing risk. Consider a problem in which the elements of the loss matrix are 
subjected to revision from time to time (such as might occur in a financial 
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application). If we know the posterior probabilities, we can trivially revise the 
minimum risk decision criterion by modifying (1.81) appropriately. If we have 
only a discriminant function, then any change to the loss matrix would require 
that we return to the training data and solve the classification problem afresh. 


Reject option. Posterior probabilities allow us to determine a rejection criterion that 
will minimize the misclassification rate, or more generally the expected loss, 
for a given fraction of rejected data points. 


Compensating for class priors. Consider our medical X-ray problem again, and 
suppose that we have collected a large number of X-ray images from the gen- 
eral population for use as training data in order to build an automated screening 
system. Because cancer is rare amongst the general population, we might find 
that, say, only 1 in every 1,000 examples corresponds to the presence of can- 
cer. If we used such a data set to train an adaptive model, we could run into 
severe difficulties due to the small proportion of the cancer class. For instance, 
a classifier that assigned every point to the normal class would already achieve 
99.9% accuracy and it would be difficult to avoid this trivial solution. Also, 
even a large data set will contain very few examples of X-ray images corre- 
sponding to cancer, and so the learning algorithm will not be exposed to a 
broad range of examples of such images and hence is not likely to generalize 
well. A balanced data set in which we have selected equal numbers of exam- 
ples from each of the classes would allow us to find a more accurate model. 
However, we then have to compensate for the effects of our modifications to 
the training data. Suppose we have used such a modified data set and found 
models for the posterior probabilities. From Bayes’ theorem (1.82), we see that 
the posterior probabilities are proportional to the prior probabilities, which we 
can interpret as the fractions of points in each class. We can therefore simply 
take the posterior probabilities obtained from our artificially balanced data set 
and first divide by the class fractions in that data set and then multiply by the 
class fractions in the population to which we wish to apply the model. Finally, 
we need to normalize to ensure that the new posterior probabilities sum to one. 
Note that this procedure cannot be applied if we have learned a discriminant 
function directly instead of determining posterior probabilities. 


Combining models. For complex applications, we may wish to break the problem 
into a number of smaller subproblems each of which can be tackled by a sep- 
arate module. For example, in our hypothetical medical diagnosis problem, 
we may have information available from, say, blood tests as well as X-ray im- 
ages. Rather than combine all of this heterogeneous information into one huge 
input space, it may be more effective to build one system to interpret the X- 
ray images and a different one to interpret the blood data. As long as each of 
the two models gives posterior probabilities for the classes, we can combine 
the outputs systematically using the rules of probability. One simple way to 
do this is to assume that, for each class separately, the distributions of inputs 
for the X-ray images, denoted by xz, and the blood data, denoted by xg, are 
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independent, so that 
p(x1,XB|Ce) = p(x1|Ce)p(xp|Cx). (1.84) 


This is an example of conditional independence property, because the indepen- 
dence holds when the distribution is conditioned on the class C. The posterior 
probability, given both the X-ray and blood data, is then given by 


p(Cx|X1, XB) ox p(X1, Xp|Cx)p(Cx) 
x  p(x1|Ce)p(xB|Cx)p(Ce) 
p(Ck|x1)p(Ck|xB) 
P(Cx) 


Thus we need the class prior probabilities p(C;,), which we can easily estimate 
from the fractions of data points in each class, and then we need to normalize 
the resulting posterior probabilities so they sum to one. The particular condi- 
tional independence assumption (1.84) is an example of the naive Bayes model. 
Note that the joint marginal distribution p(x;, xg) will typically not factorize 
under this model. We shall see in later chapters how to construct models for 
combining data that do not require the conditional independence assumption 
(1.84). 


(1.85) 


1.5.5 Loss functions for regression 


So far, we have discussed decision theory in the context of classification prob- 
lems. We now turn to the case of regression problems, such as the curve fitting 
example discussed earlier. The decision stage consists of choosing a specific esti- 
mate y(x) of the value of ¢ for each input x. Suppose that in doing so, we incur a 
loss L(t, y(x)). The average, or expected, loss is then given by 


E[L] = 1 L(t, y(x))p(x, t) dx dt. (1.86) 


A common choice of loss function in regression problems is the squared loss given 
by L(t, y(x)) = {y(x) — t}. In this case, the expected loss can be written 


E[L] = I {y(x) — t} p(x, t) dxdt. (1.87) 


Our goal is to choose y(x) so as to minimize E[L]. If we assume a completely 
flexible function y(x), we can do this formally using the calculus of variations to 
give 


ELL] _ x) z _ 
me) =2 ftv ) — thp(x, t) dt = 0. (1.88) 


Solving for y(x), and using the sum and product rules of probability, we obtain 


f tp(x, t) dt 


y(x) = = = [es dt = E;[t|x] (1.89) 


Figure 1.28 The regression function y(x), 


Exercise 1.25 
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which minimizes the expected 
squared loss, is given by the 
mean of the conditional distri- 
bution p(t|z). 


XO x 


which is the conditional average of t conditioned on x and is known as the regression 
function. This result is illustrated in Figure 1.28. It can readily be extended to mul- 
tiple target variables represented by the vector t, in which case the optimal solution 
is the conditional average y(x) = E;[t|x]. 

We can also derive this result in a slightly different way, which will also shed 
light on the nature of the regression problem. Armed with the knowledge that the 
optimal solution is the conditional expectation, we can expand the square term as 
follows 


{y(x) — t} = {y(x) — Efe|x] + Eft|x] — t} 
= {y(x) — Eft[x]}’ + 2{y(x) — E[t}x]}{E[é|x] — t} + {E[é[x] — t}? 
where, to keep the notation uncluttered, we use E[t|x] to denote E,[t|x]. Substituting 


into the loss function and performing the integral over t, we see that the cross-term 
vanishes and we obtain an expression for the loss function in the form 


BIL) = | (u(x) - EE x) ax-+ fte 


The function y(x) we seek to determine enters only in the first term, which will be 
minimized when y(x) is equal to E[¢|x], in which case this term will vanish. This 
is simply the result that we derived previously and that shows that the optimal least 
squares predictor is given by the conditional mean. The second term is the variance 
of the distribution of t, averaged over x. It represents the intrinsic variability of 
the target data and can be regarded as noise. Because it is independent of y(x), it 
represents the irreducible minimum value of the loss function. 

As with the classification problem, we can either determine the appropriate prob- 
abilities and then use these to make optimal decisions, or we can build models that 
make decisions directly. Indeed, we can identify three distinct approaches to solving 
regression problems given, in order of decreasing complexity, by: 


cae 


x] — t}°p(x) dx. (1.90) 


(a) First solve the inference problem of determining the joint density p(x, t). Then 
normalize to find the conditional density p(t|x), and finally marginalize to find 
the conditional mean given by (1.89). 
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(b) First solve the inference problem of determining the conditional density p(t|x), 
and then subsequently marginalize to find the conditional mean given by (1.89). 


(c) Find a regression function y(x) directly from the training data. 


The relative merits of these three approaches follow the same lines as for classifica- 
tion problems above. 

The squared loss is not the only possible choice of loss function for regression. 
Indeed, there are situations in which squared loss can lead to very poor results and 
where we need to develop more sophisticated approaches. An important example 
concerns situations in which the conditional distribution p(t|x) is multimodal, as 
often arises in the solution of inverse problems. Here we consider briefly one simple 
generalization of the squared loss, called the Minkowski loss, whose expectation is 
given by 


E[La] = J| |y(x) — t|1p(x, t) dx dt (1.91) 


which reduces to the expected squared loss for q = 2. The function |y — t|4 is 
plotted against y — t for various values of q in Figure 1.29. The minimum of E[L,] 
is given by the conditional mean for q = 2, the conditional median for q = 1, and 
the conditional mode for q — 0. 


Information Theory 


In this chapter, we have discussed a variety of concepts from probability theory and 
decision theory that will form the foundations for much of the subsequent discussion 
in this book. We close this chapter by introducing some additional concepts from 
the field of information theory, which will also prove useful in our development of 
pattern recognition and machine learning techniques. Again, we shall focus only on 
the key concepts, and we refer the reader elsewhere for more detailed discussions 
(Viterbi and Omura, 1979; Cover and Thomas, 1991; MacKay, 2003) . 

We begin by considering a discrete random variable x and we ask how much 
information is received when we observe a specific value for this variable. The 
amount of information can be viewed as the ‘degree of surprise’ on learning the 
value of x. If we are told that a highly improbable event has just occurred, we will 
have received more information than if we were told that some very likely event 
has just occurred, and if we knew that the event was certain to happen we would 
receive no information. Our measure of information content will therefore depend 
on the probability distribution p(x), and we therefore look for a quantity h(a) that 
is a monotonic function of the probability p(x) and that expresses the information 
content. The form of h(-) can be found by noting that if we have two events x 
and y that are unrelated, then the information gain from observing both of them 
should be the sum of the information gained from each of them separately, so that 
h(x, y) = h(a) + h(y). Two unrelated events will be statistically independent and 
so p(x, y) = p(x)p(y). From these two relationships, it is easily shown that h(x) 
must be given by the logarithm of p(x) and so we have 
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ly — t| 


Figure 1.29 Plots of the quantity L4 = |y — t|? for various values of q. 


h(x) = — log, p(x) (1.92) 


where the negative sign ensures that information is positive or zero. Note that low 
probability events x correspond to high information content. The choice of basis 
for the logarithm is arbitrary, and for the moment we shall adopt the convention 
prevalent in information theory of using logarithms to the base of 2. In this case, as 
we shall see shortly, the units of h(x) are bits (‘binary digits’). 

Now suppose that a sender wishes to transmit the value of a random variable to 
a receiver. The average amount of information that they transmit in the process is 
obtained by taking the expectation of (1.92) with respect to the distribution p(x) and 
is given by 


H[z] = — S > p(x) log, p(x). (1.93) 


This important quantity is called the entropy of the random variable x. Note that 
lim, —.9 pln p = 0 and so we shall take p(x) In p(x) = 0 whenever we encounter a 
value for x such that p(x) = 0. 

So far we have given a rather heuristic motivation for the definition of informa- 
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tion (1.92) and the corresponding entropy (1.93). We now show that these definitions 
indeed possess useful properties. Consider a random variable x having 8 possible 
states, each of which is equally likely. In order to communicate the value of x to 
a receiver, we would need to transmit a message of length 3 bits. Notice that the 
entropy of this variable is given by 


1 1 . 
H[z] = —8 x log, = 3 bits. 
Now consider an example (Cover and Thomas, 1991) of a variable having 8 pos- 
sible states {a, b,c, d, e, f, g, h} for which the respective probabilities are given by 


111d 1 1 
(Sa a oe The entropy in this case is given by 


1 


1 1 1 ; 
Ale] = 3 log, ~i logs 7 = 2 bits. 


Lio 1 lo eas 1 
82 3 — 16 “5216 64 2 64 


8 
We see that the nonuniform distribution has a smaller entropy than the uniform one, 
and we shall gain some insight into this shortly when we discuss the interpretation of 
entropy in terms of disorder. For the moment, let us consider how we would transmit 
the identity of the variable’s state to a receiver. We could do this, as before, using 
a 3-bit number. However, we can take advantage of the nonuniform distribution by 
using shorter codes for the more probable events, at the expense of longer codes for 
the less probable events, in the hope of getting a shorter average code length. This 
can be done by representing the states {a, b, c, d,e, f, g, h} using, for instance, the 
following set of code strings: 0, 10, 110, 1110, 111100, 111101, 111110, 111111. 
The average length of the code that has to be transmitted is then 


average code length = ! x14 : x 24 : x 34 = x4+4~x = x 6 = 2 bits 
which again is the same as the entropy of the random variable. Note that shorter code 
strings cannot be used because it must be possible to disambiguate a concatenation 
of such strings into its component parts. For instance, 11001110 decodes uniquely 
into the state sequence c, a, d. 

This relation between entropy and shortest coding length is a general one. The 
noiseless coding theorem (Shannon, 1948) states that the entropy is a lower bound 
on the number of bits needed to transmit the state of a random variable. 

From now on, we shall switch to the use of natural logarithms in defining en- 
tropy, as this will provide a more convenient link with ideas elsewhere in this book. 
In this case, the entropy is measured in units of ‘nats’ instead of bits, which differ 
simply by a factor of ln 2. 

We have introduced the concept of entropy in terms of the average amount of 
information needed to specify the state of a random variable. In fact, the concept of 
entropy has much earlier origins in physics where it was introduced in the context 
of equilibrium thermodynamics and later given a deeper interpretation as a measure 
of disorder through developments in statistical mechanics. We can understand this 
alternative view of entropy by considering a set of N identical objects that are to be 
divided amongst a set of bins, such that there are n; objects in the it” bin. Consider 
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the number of different ways of allocating the objects to the bins. There are N 
ways to choose the first object, (N — 1) ways to choose the second object, and 
so on, leading to a total of N! ways to allocate all N objects to the bins, where N! 
(pronounced ‘factorial N’) denotes the product N x (IV —1) x---x2x 1. However, 
we don’t wish to distinguish between rearrangements of objects within each bin. In 
the it bin there are n;! ways of reordering the objects, and so the total number of 
ways of allocating the N objects to the bins is given by 

wa (1.94) 

IL n! 

which is called the multiplicity. The entropy is then defined as the logarithm of the 
multiplicity scaled by an appropriate constant 


H = mw- In N! yam (1.95) 


We now consider the limit N — 00, in which the fractions n;/N are held fixed, and 
apply Stirling’s approximation 


InN!~ Ninn N-N (1.96) 


: Ni Ni i l 
o (7) my) = > _Pilnpi ont 


where we have used `; n; = N. Here pj = limy—o(ni/N) is the probability 
of an object being assigned to the it bin. In physics terminology, the specific ar- 
rangements of objects in the bins is called a microstate, and the overall distribution 
of occupation numbers, expressed through the ratios n;/N, is called a macrostate. 
The multiplicity W is also known as the weight of the macrostate. 

We can interpret the bins as the states x; of a discrete random variable X, where 
p(X = zi) = p;. The entropy of the random variable X is then 


which gives 


— S$ p(z:) n p(z). (1.98) 


Distributions p(;) that are sharply peaked around a few values will have a relatively 
low entropy, whereas those that are spread more evenly across pi values will 
have higher entropy, as illustrated in Figure 1.30. Because 0 < p; < 1, the entropy 
is nonnegative, and it will equal its minimum value of O when one of the p; = 
1 and all other p;z; = 0. The maximum entropy configuration can be found by 
maximizing H using a Lagrange multiplier to enforce the normalization constraint 
on the probabilities. Thus we maximize 


H=- > P(e) mp(z:) FA (Sre = 1) (1.99) 
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Figure 1.30 Histograms of two probability distributions over 30 bins illustrating the higher value of the entropy 
H for the broader distribution. The largest entropy would arise from a uniform distribution that would give H = 


—In(1/30) = 3.40. 


Exercise 1.29 


from which we find that all of the p(x;) are equal and are given by p(x;) = 1/M 
where M is the total number of states x;. The corresponding value of the entropy 
is then H = In M. This result can also be derived from Jensen’s inequality (to be 
discussed shortly). To verify that the stationary point is indeed a maximum, we can 
evaluate the second derivative of the entropy, which gives 


ou 1 
= —I;; (1.100) 
Op(x;)Op(x;) 1D; 


where J; are the elements of the identity matrix. 

We can extend the definition of entropy to include distributions p(x) over con- 
tinuous variables x as follows. First divide x into bins of width A. Then, assuming 
p(x) is continuous, the mean value theorem (Weisstein, 1999) tells us that, for each 
such bin, there must exist a value x; such that 


(i+1)A 
l p(x)dz = p(x;)A. (1.101) 
iA 

We can now quantize the continuous variable x by assigning any value z to the value 


xi whenever x falls in the i” bin. The probability of observing the value z; is then 
p(a;)A. This gives a discrete distribution for which the entropy takes the form 


Ha S p(z:;)Aln (p(z:i)A) = — S © p(a:)A In p(2i) -hA (1.102) 


where we have used }°, p(a;)A = 1, which follows from (1.101). We now omit 
the second term — In A on the right-hand side of (1.102) and then consider the limit 
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A — 0. The first term on the right-hand side of (1.102) will approach the integral of 
p(x) In p(x) in this limit so that 


lim 
A—-0 


Yi rle)Ainple,) | == f ple) impede (1103) 


where the quantity on the right-hand side is called the differential entropy. We see 
that the discrete and continuous forms of the entropy differ by a quantity In A, which 
diverges in the limit A — 0. This reflects the fact that to specify a continuous 
variable very precisely requires a large number of bits. For a density defined over 
multiple continuous variables, denoted collectively by the vector x, the differential 
entropy is given by 


H|x] = — frw ln p(x) dx. (1.104) 

In the case of discrete distributions, we saw that the maximum entropy con- 
figuration corresponded to an equal distribution of probabilities across the possible 
states of the variable. Let us now consider the maximum entropy configuration for 
a continuous variable. In order for this maximum to be well defined, it will be nec- 
essary to constrain the first and second moments of p(x) as well as preserving the 
normalization constraint. We therefore maximize the differential entropy with the 


Ludwig Boltzmann 
1844—1906 


Ludwig Eduard Boltzmann was an 
Austrian physicist who created the 
field of statistical mechanics. Prior 
to Boltzmann, the concept of en- 
tropy was already known from 
classical thermodynamics where it 
quantifies the fact that when we take energy from a 
system, not all of that energy is typically available 
to do useful work. Boltzmann showed that the ther- 
modynamic entropy S, a macroscopic quantity, could 
be related to the statistical properties at the micro- 
scopic level. This is expressed through the famous 
equation S = klnW in which W represents the 
number of possible microstates in a macrostate, and 
k ~ 1.38 x 107” (in units of Joules per Kelvin) is 
known as Boltzmann’s constant. Boltzmann’s ideas 
were disputed by many scientists of they day. One dif- 
ficulty they saw arose from the second law of thermo- 


dynamics, which states that the entropy of a closed 
system tends to increase with time. By contrast, at 
the microscopic level the classical Newtonian equa- 
tions of physics are reversible, and so they found it 
difficult to see how the latter could explain the for- 
mer. They didn’t fully appreciate Boltzmann’s argu- 
ments, which were statistical in nature and which con- 
cluded not that entropy could never decrease over 
time but simply that with overwhelming probability it 
would generally increase. Boltzmann even had a long- 
running dispute with the editor of the leading German 
physics journal who refused to let him refer to atoms 
and molecules as anything other than convenient the- 
oretical constructs. The continued attacks on his work 
lead to bouts of depression, and eventually he com- 
mitted suicide. Shortly after Boltzmann’s death, new 
experiments by Perrin on colloidal suspensions veri- 
fied his theories and confirmed the value of the Boltz- 
mann constant. The equation S = k ln W is carved on 
Boltzmann’s tombstone. 
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Exercise 1.35 


three constraints 


f p(z)dzr = 1 (1.105) 
f zp(x)dr = u (1.106) 
f (x— 1)*p(x)dz = o°. (1.107) 


The constrained maximization can be performed using Lagrange multipliers so that 
we maximize the following functional with respect to p(x) 


- f v(e)mpleyae + rs (| 2) ar -1) 
+Az2 e rp(x) dx — n) + A3 (Je — p)*p(x) dx — a°) . 


Using the calculus of variations, we set the derivative of this functional to zero giving 


p(x) = exp { 1+ Ai + A22 + A3(x uy}. (1.108) 


The Lagrange multipliers can be found by back substitution of this result into the 
three constraint equations, leading finally to the result 


1 _ 2 
Pa) = Basar exp { ae } (1.109) 


and so the distribution that maximizes the differential entropy is the Gaussian. Note 
that we did not constrain the distribution to be nonnegative when we maximized the 
entropy. However, because the resulting distribution is indeed nonnegative, we see 
with hindsight that such a constraint is not necessary. 

If we evaluate the differential entropy of the Gaussian, we obtain 


H{z] = ; {1+ In(270°)}. (1.110) 


Thus we see again that the entropy increases as the distribution becomes broader, 
i.e., as 07 increases. This result also shows that the differential entropy, unlike the 
discrete entropy, can be negative, because H(x) < 0 in (1.110) for o? < 1/(27e). 

Suppose we have a joint distribution p(x, y) from which we draw pairs of values 
of x and y. If a value of x is already known, then the additional information needed 
to specify the corresponding value of y is given by — ln p(y|x). Thus the average 
additional information needed to specify y can be written as 


Hly|x] = - f] vy.) mplybo ay ax (1.111) 
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which is called the conditional entropy of y given x. It is easily seen, using the 
product rule, that the conditional entropy satisfies the relation 

H[x, y] = Hly|x] + H[x] (1.112) 
where H[x, y] is the differential entropy of p(x, y) and H[x] is the differential en- 
tropy of the marginal distribution p(x). Thus the information needed to describe x 
and y is given by the sum of the information needed to describe x alone plus the 
additional information required to specify y given x. 


1.6.1 Relative entropy and mutual information 


So far in this section, we have introduced a number of concepts from information 
theory, including the key notion of entropy. We now start to relate these ideas to 
pattern recognition. Consider some unknown distribution p(x), and suppose that 
we have modelled this using an approximating distribution q(x). If we use q(x) to 
construct a coding scheme for the purpose of transmitting values of x to a receiver, 
then the average additional amount of information (in nats) required to specify the 
value of x (assuming we choose an efficient coding scheme) as a result of using q(x) 
instead of the true distribution p(x) is given by 


= f IA (- f p(x) In p(x) ax) 
e - [ond | ae. 


This is known as the relative entropy or Kullback-Leibler divergence, or KL diver- 
gence (Kullback and Leibler, 1951), between the distributions p(x) and q(x). Note 
that it is not a symmetrical quantity, that is to say KL(p||q) 4 KL(q||p). 

We now show that the Kullback-Leibler divergence satisfies KL(p||q) > 0 with 
equality if, and only if, p(x) = q(x). To do this we first introduce the concept of 
convex functions. A function f(x) is said to be convex if it has the property that 
every chord lies on or above the function, as shown in Figure 1.31. Any value of x 
in the interval from x = a to x = b can be written in the form Aa + (1 — A)b where 
0 < à < 1. The corresponding point on the chord is given by A f(a) + (1 — A) f(b), 


KL(p||q) 


(1.113) 


= Claude Shannon 
me 1916-2001 


"ay After graduating from Michigan and 
MIT, Shannon joined the AT&T Bell 
Telephone laboratories in 1941. His 
paper ‘A Mathematical Theory of 
Communication’ published in the 
Bell System Technical Journal in 
1948 laid the foundations for modern information the- 


ory. This paper introduced the word ‘bit’, and his con- 
cept that information could be sent as a stream of 1s 
and Os paved the way for the communications revo- 
lution. It is said that von Neumann recommended to 
Shannon that he use the term entropy, not only be- 
cause of its similarity to the quantity used in physics, 
but also because “nobody knows what entropy really 
is, SO in any discussion you will always have an advan- 
tage”. 
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Figure 1.31 A convex function f(x) is one for which ev- 
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ery chord (shown in blue) lies on or above 
the function (shown in red). 


QP----------- 
Qo }e------- - -- 


TA 


and the corresponding value of the function is f (Aa + (1 — A)b). Convexity then 
implies 
f(Aa+ (1 — A)b) < Af (a) + (1 — A) f(b). (1.114) 

This is equivalent to the requirement that the second derivative of the function be 
everywhere positive. Examples of convex functions are x In x (for x > 0) and z?. A 
function is called strictly convex if the equality is satisfied only for A = 0 and A = 1. 
If a function has the opposite property, namely that every chord lies on or below the 
function, it is called concave, with a corresponding definition for strictly concave. If 
a function f(x) is convex, then — f(x) will be concave. 

Using the technique of proof by induction, we can show from (1.114) that a 
convex function f(x) satisfies 


M M 
f (>: xas) < So Ada (1.115) 
i=1 t=1 


where À; > 0 and `; A; = 1, for any set of points {x;}. The result (1.115) is 
known as Jensen’s inequality. If we interpret the À; as the probability distribution 
over a discrete variable x taking the values {x;}, then (1.115) can be written 


f(Elz]) < E[f(2)] (1.116) 


where E|-] denotes the expectation. For continuous variables, Jensen’s inequality 
takes the form 


f ( / xp(x) ax) < | f()p(x) dx. (1.117) 


We can apply Jensen’s inequality in the form (1.117) to the Kullback-Leibler 
divergence (1.113) to give 


KL(pllq) = - f oem { 9} dx > -im f a(x) dx=0 (1.118) 
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where we have used the fact that — ln x is a convex function, together with the nor- 
malization condition f q(x)dx = 1. In fact, — ln is a strictly convex function, 
so the equality will hold if, and only if, g(x) = p(x) for all x. Thus we can in- 
terpret the Kullback-Leibler divergence as a measure of the dissimilarity of the two 
distributions p(x) and q(x). 

We see that there is an intimate relationship between data compression and den- 
sity estimation (i.e., the problem of modelling an unknown probability distribution) 
because the most efficient compression is achieved when we know the true distri- 
bution. If we use a distribution that is different from the true one, then we must 
necessarily have a less efficient coding, and on average the additional information 
that must be transmitted is (at least) equal to the Kullback-Leibler divergence be- 
tween the two distributions. 

Suppose that data is being generated from an unknown distribution p(x) that we 
wish to model. We can try to approximate this distribution using some parametric 
distribution q(x|@), governed by a set of adjustable parameters 0, for example a 
multivariate Gaussian. One way to determine 0 is to minimize the Kullback-Leibler 
divergence between p(x) and q(x|@) with respect to 0. We cannot do this directly 
because we don’t know p(x). Suppose, however, that we have observed a finite set 
of training points Xn, for n = 1,...,N, drawn from p(x). Then the expectation 
with respect to p(x) can be approximated by a finite sum over these points, using 
(1.35), so that 

N 
KL(p|lq) ~ X- {—In g(x, |9) + m p(xn)} (1.119) 
n=1 
The second term on the right-hand side of (1.119) is independent of 0, and the first 
term is the negative log likelihood function for 0 under the distribution q(x|@) eval- 
uated using the training set. Thus we see that minimizing this Kullback-Leibler 
divergence is equivalent to maximizing the likelihood function. 

Now consider the joint distribution between two sets of variables x and y given 
by p(x, y). If the sets of variables are independent, then their joint distribution will 
factorize into the product of their marginals p(x, y) = p(x)p(y). If the variables are 
not independent, we can gain some idea of whether they are ‘close’ to being indepen- 
dent by considering the Kullback-Leibler divergence between the joint distribution 
and the product of the marginals, given by 


p(x, y)llpx)p(y)) 


“fs p(x, y yin(# pn %2) dx dy (1.120) 


which is called the mutual information between the variables x and y. From the 
properties of the Kullback-Leibler divergence, we see that I(x, y) > 0 with equal- 
ity if, and only if, x and y are independent. Using the sum and product rules of 
probability, we see that the mutual information is related to the conditional entropy 
through 


Ill 


I[x, y] 


I[x, y] = H[x] — H[x|y] = H[y] — H[y|x]. (1.121) 
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Thus we can view the mutual information as the reduction in the uncertainty about x 
by virtue of being told the value of y (or vice versa). From a Bayesian perspective, 
we can view p(x) as the prior distribution for x and p(x|y) as the posterior distribu- 
tion after we have observed new data y. The mutual information therefore represents 
the reduction in uncertainty about x as a consequence of the new observation y. 


(x) FQ Consider the sum-of-squares error function given by (1.2) in which 
the function y(x,w) is given by the polynomial (1.1). Show that the coefficients 
w = {w;} that minimize this error function are given by the solution to the following 
set of linear equations 


M 
D Azt = T; (1.122) 
j=0 
where 
N N 
Age any T=) Ga) (1.123) 
n=1 n=1 


Here a suffix i or j denotes the index of a component, whereas ()' denotes x raised 
to the power of i. 


(x) Write down the set of coupled linear equations, analogous to (1.122), satisfied 
by the coefficients w; which minimize the regularized sum-of-squares error function 
given by (1.4). 


(xx) Suppose that we have three coloured boxes r (red), b (blue), and g (green). 
Box r contains 3 apples, 4 oranges, and 3 limes, box b contains 1 apple, 1 orange, 
and 0 limes, and box g contains 3 apples, 3 oranges, and 4 limes. If a box is chosen 
at random with probabilities p(r) = 0.2, p(b) = 0.2, p(g) = 0.6, and a piece of 
fruit is removed from the box (with equal probability of selecting any of the items in 
the box), then what is the probability of selecting an apple? If we observe that the 
selected fruit is in fact an orange, what is the probability that it came from the green 
box? 


œ» FY Consider a probability density p,(x) defined over a continuous vari- 
able x, and suppose that we make a nonlinear change of variable using x = g(y), 
so that the density transforms according to (1.27). By differentiating (1.27), show 
that the location 7 of the maximum of the density in y is not in general related to the 
location £ of the maximum of the density over x by the simple functional relation 
T = g(y) as a consequence of the Jacobian factor. This shows that the maximum 
of a probability density (in contrast to a simple function) is dependent on the choice 
of variable. Verify that, in the case of a linear transformation, the location of the 
maximum transforms in the same way as the variable itself. 


(x) Using the definition (1.38) show that var[f(2)] satisfies (1.39). 


1.6 


1.7 


1.8 


1.9 
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(x) Show that if two variables x and y are independent, then their covariance is 
zero. 


œx» [TY = In this exercise, we prove the normalization condition (1.48) for the 
univariate Gaussian. To do this consider, the integral 


a 1 
r= | exp (- 53”) dx (1.124) 


which we can evaluate by first writing its square in the form 


2 2 2 
P= J. D exp (z — 5524 ) da dy. (1.125) 


Now make the transformation from Cartesian coordinates (x, y) to polar coordinates 
(r,@) and then substitute u = r?. Show that, by performing the integrals over 6 and 
u, and then taking the square root of both sides, we obtain 


I = (2r). (1.126) 


Finally, use this result to show that the Gaussian distribution N (x| u, 07) is normal- 
ized. 


œ» PUY By using a change of variables, verify that the univariate Gaussian 
distribution given by (1.46) satisfies (1.49). Next, by differentiating both sides of the 
normalization condition 


f N (a|p,07) dz =1 (1.127) 


—Co 


with respect to o”, verify that the Gaussian satisfies (1.50). Finally, show that (1.51) 
holds. 


œ FQ Show that the mode (i.e. the maximum) of the Gaussian distribution 
(1.46) is given by u. Similarly, show that the mode of the multivariate Gaussian 
(1.52) is given by ps. 


(x) [ZY Suppose that the two variables x and z are statistically independent. 
Show that the mean and variance of their sum satisfies 


Eje +z] = Eļz] + Elz] (1.128) 
varje +z] = vara] + var[z]. (1.129) 


(x) By setting the derivatives of the log likelihood function (1.54) with respect to u 
and o? equal to zero, verify the results (1.55) and (1.56). 
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œ» FY Using the results (1.49) and (1.50), show that 


Eltptnl = + Inmo” (1.130) 


where £n and £m denote data points sampled from a Gaussian distribution with mean 
u and variance o°, and Ipm satisfies Inm = 1 if n = m and Inm = 0 otherwise. 
Hence prove the results (1.57) and (1.58). 


(x) Suppose that the variance of a Gaussian is estimated using the result (1.56) but 
with the maximum likelihood estimate um, replaced with the true value u of the 
mean. Show that this estimator has the property that its expectation is given by the 
true variance o?. 


(xx) Show that an arbitrary square matrix with elements w;; can be written in 
the form wij = wy, + wi where Wi; and wi are symmetric and anti-symmetric 
matrices, respectively, satisfying ws, = ws, and wi = -w for all ¿ and j. Now 
consider the second order term in a higher order polynomial in D dimensions, given 


by 
D D 
eerie: (1.131) 
i=1 j=l 
Show that 
D D D D 
>S apes = S 7 weary (1.132) 
i=1 j=l i=1 j=1 


so that the contribution from the anti-symmetric matrix vanishes. We therefore see 
that, without loss of generality, the matrix of coefficients w;; can be chosen to be 
symmetric, and so not all of the D? elements of this matrix can be chosen indepen- 
dently. Show that the number of independent parameters in the matrix wi is given 
by D(D + 1)/2. 


(x * x) FQ In this exercise and the next, we explore how the number of indepen- 
dent parameters in a polynomial grows with the order M of the polynomial and with 
the dimensionality D of the input space. We start by writing down the M‘ order 
term for a polynomial in D dimensions in the form 


D D D 
» p3 En >. Wiiiz im Titia’ Ving: (1.133) 


44=1 ig=1 im=l1 


The coefficients w;,j,...:,, comprise D™ elements, but the number of independent 
parameters is significantly fewer due to the many interchange symmetries of the 
factor £i, Li, °-- Ziu. Begin by showing that the redundancy in the coefficients can 
be removed by rewriting this M*} order term in the form 


imM-1 


D i 
>, a >D Wiriz--im Ti Tia t Tim: (1.134) 


i=l ip=1 im=1 
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Note that the precise relationship between the w coefficients and w coefficients need 
not be made explicit. Use this result to show that the number of independent param- 
eters n( D, M), which appear at order M, satisfies the following recursion relation 


D 
n(D,M) =X nli, M -1). (1.135) 
i=1 


Next use proof by induction to show that the following result holds 


= (1.136) 


D 
(i+M —2)! (D+ M-—1)! 
Gi -~  (D—1)!M! 


which can be done by first proving the result for D = 1 and arbitrary M by making 
use of the result 0! = 1, then assuming it is correct for dimension D and verifying 


that it is correct for dimension D + 1. Finally, use the two previous results, together 
with proof by induction, to show 


(D+ M —-1)! 
(D—1)!M!} ° 

To do this, first show that the result is true for W = 2, and any value of D > 1, 

by comparison with the result of Exercise 1.14. Then make use of (1.135), together 


with (1.136), to show that, if the result holds at order M — 1, then it will also hold at 
order M 


n(D, M) = (1.137) 


(xxx) In Exercise 1.15, we proved the result (1.135) for the number of independent 
parameters in the M*® order term of a D-dimensional polynomial. We now find an 
expression for the total number N(D, M) of independent parameters in all of the 
terms up to and including the M6th order. First show that N(D, M) satisfies 


M 
N(D,M) = X` n(D,m) (1.138) 
m=0 
where n(D,m) is the number of independent parameters in the term of order m. 
Now make use of the result (1.137), together with proof by induction, to show that 
_ (D+M)! 
N(M) = Drm! 
This can be done by first proving that the result holds for M = 0 and arbitrary 


D > 1, then assuming that it holds at order M, and hence showing that it holds at 
order M + 1. Finally, make use of Stirling’s approximation in the form 


ninene (1.140) 


for large n to show that, for D >> M, the quantity N(D, M) grows like D™, 
and for M > D it grows like MP. Consider a cubic (M = 3) polynomial in D 
dimensions, and evaluate numerically the total number of independent parameters 
for G) D = 10 and Gi) D = 100, which correspond to typical small-scale and 
medium-scale machine learning applications. 


(1.139) 
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œ» FQ The gamma function is defined by 


T(x) =f u” te~" du. (1.141) 
0 


Using integration by parts, prove the relation T(x + 1) = zT (x). Show also that 
T(1) = 1 and hence that T(x + 1) = x! when z is an integer. 


(« x) | www | We can use the result (1.126) to derive an expression for the surface 
area Sp, and the volume Vp, of a sphere of unit radius in D dimensions. To do this, 
consider the following result, which is obtained by transforming from Cartesian to 
polar coordinates 


D GÒ co 
IL / e7% dr; = Sp | en” rP- dr. (1.142) 
i=1 Y 709 0 


Using the definition (1.141) of the Gamma function, together with (1.126), evaluate 
both sides of this equation, and hence show that 


QP /2 


Next, by integrating with respect to radius from 0 to 1, show that the volume of the 
unit sphere in D dimensions is given by 
= 


Finally, use the results [(1) = 1 and ['(3/2) = y7/2 to show that (1.143) and 
(1.144) reduce to the usual expressions for D = 2 and D = 3. 


Vp (1.144) 


(xx) Consider a sphere of radius a in D-dimensions together with the concentric 
hypercube of side 2a, so that the sphere touches the hypercube at the centres of each 
of its sides. By using the results of Exercise 1.18, show that the ratio of the volume 
of the sphere to the volume of the cube is given by 


volume of sphere _ ‘ad l (1.145) 
volume of cube D2P-1T(D/2) 
Now make use of Stirling’s formula in the form 
T(x +1) œ (20) 2-2 41/2 (1.146) 


which is valid for x >> 1, to show that, as D — ov, the ratio (1.145) goes to zero. 
Show also that the ratio of the distance from the centre of the hypercube to one of 
the corners, divided by the perpendicular distance to one of the sides, is VD, which 
therefore goes to co as D — oo. From these results we see that, in a space of high 
dimensionality, most of the volume of a cube is concentrated in the large number of 
corners, which themselves become very long ‘spikes’! 
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(« x) EAA In this exercise, we explore the behaviour of the Gaussian distribution 
in high-dimensional spaces. Consider a Gaussian distribution in D dimensions given 
by 
ii aes ell) (1.147) 
(2ro?)P/2 20? 


We wish to find the density with respect to radius in polar coordinates in which the 
direction variables have been integrated out. To do this, show that the integral of 
the probability density over a thin shell of radius r and thickness e€, where « « 1, is 


given by p(r)e where 
Sot? r? 
PW") = ropa P (aa (1.148) 


where Sp is the surface area of a unit sphere in D dimensions. Show that the function 
p(r) has a single stationary point located, for large D, at? ~ v Dø. By considering 
p(T + €) where e <7, show that for large D, 


A e 3e? 
p(T + €) = p(T) exp ( = (1.149) 
which shows that 7 is a maximum of the radial probability density and also that p(r) 
decays exponentially away from its maximum at 7 with length scale o. We have 
already seen that o < T for large D, and so we see that most of the probability 
mass is concentrated in a thin shell at large radius. Finally, show that the probability 
density p(x) is larger at the origin than at the radius 7 by a factor of exp(D/2). 
We therefore see that most of the probability mass in a high-dimensional Gaussian 
distribution is located at a different radius from the region of high probability density. 
This property of distributions in spaces of high dimensionality will have important 
consequences when we consider Bayesian inference of model parameters in later 
chapters. 


(xx) Consider two nonnegative numbers a and b, and show that, if a < b, then 
a < (ab)'/?. Use this result to show that, if the decision regions of a two-class 
classification problem are chosen to minimize the probability of misclassification, 
this probability will satisfy 


p(mistake) < J repe. dx. (1.150) 


(x) [Y= Given a loss matrix with elements Lpz, the expected risk is minimized 
if, for each x, we choose the class that minimizes (1.81). Verify that, when the 
loss matrix is given by Lgj = 1 — Iķj, where Ip; are the elements of the identity 
matrix, this reduces to the criterion of choosing the class having the largest posterior 
probability. What is the interpretation of this form of loss matrix? 


(x) Derive the criterion for minimizing the expected loss when there is a general 
loss matrix and general prior probabilities for the classes. 
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1.24 


1.25 


1.26 


1.27 


1.28 


1.29 


1.30 


œ» PQQ Consider a classification problem in which the loss incurred when 
an input vector from class Cx is classified as belonging to class C; is given by the 
loss matrix L;;, and for which the loss incurred in selecting the reject option is À. 
Find the decision criterion that will give the minimum expected loss. Verify that this 
reduces to the reject criterion discussed in Section 1.5.3 when the loss matrix is given 
by Ly; = 1 — I,;. What is the relationship between A and the rejection threshold 0? 


(x) [QQ Consider the generalization of the squared loss function (1.87) for a 
single target variable t to the case of multiple target variables described by the vector 
t given by 


E[L(t, y(x))] = | / v(x) — t?p(x, t) dxdt. (1.151) 


Using the calculus of variations, show that the function y(x) for which this expected 
loss is minimized is given by y(x) = E¢[t|x]. Show that this result reduces to (1.89) 
for the case of a single target variable t. 


(x) By expansion of the square in (1.151), derive a result analogous to (1.90) and 
hence show that the function y(x) that minimizes the expected squared loss for the 
case of a vector t of target variables is again given by the conditional expectation of 
t. 


œ» QTY Consider the expected loss for regression problems under the Lq loss 
function given by (1.91). Write down the condition that y(x) must satisfy in order 
to minimize E[LZ,]. Show that, for q = 1, this solution represents the conditional 
median, i.e., the function y(x) such that the probability mass for t < y(x) is the 
same as for t > y(x). Also show that the minimum expected Lq loss for q — 0 is 
given by the conditional mode, i.e., by the function y(x) equal to the value of t that 
maximizes p(t|x) for each x. 


(x) In Section 1.6, we introduced the idea of entropy h(x) as the information gained 
on observing the value of a random variable x having distribution p(x). We saw 
that, for independent variables x and y for which p(x, y) = p(a)p(y), the entropy 
functions are additive, so that h(x, y) = h(a) + h(y). In this exercise, we derive the 
relation between h and p in the form of a function h(p). First show that h(p?) = 
2h(p), and hence by induction that h(p”) = nh(p) where n is a positive integer. 
Hence show that h(p"/™) = (n/m)h(p) where m is also a positive integer. This 
implies that h(p”) = xh(p) where x is a positive rational number, and hence by 
continuity when it is a positive real number. Finally, show that this implies h(p) 
must take the form h(p) œ ln p. 


(x) FY Consider an M-state discrete random variable x, and use Jensen’s in- 
equality in the form (1.115) to show that the entropy of its distribution p(x) satisfies 
H[z] < ln M. 


(xx) Evaluate the Kullback-Leibler divergence (1.113) between two Gaussians 


p(x) = N(z|u, 07) and q(x) = N(2|m, s”). 
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The joint distribution p(x, y) for two binary variables 
x and y used in Exercise 1.39. 


œ» EX Consider two variables x and y having joint distribution p(x, y). Show 
that the differential entropy of this pair of variables satisfies 


H[x, y] < H[x] + Hy] (1.152) 
with equality if, and only if, x and y are statistically independent. 


(x) Consider a vector x of continuous variables with distribution p(x) and corre- 
sponding entropy H[x]. Suppose that we make a nonsingular linear transformation 
of x to obtain a new variable y = Ax. Show that the corresponding entropy is given 
by H[y] = H[x] + ln |A| where |A| denotes the determinant of A. 


(xx) Suppose that the conditional entropy H[y|z] between two discrete random 
variables x and y is zero. Show that, for all values of x such that p(x) > 0, the 
variable y must be a function of x, in other words for each zx there is only one value 
of y such that p(y|a) A 0. 


œ» FQ Use the calculus of variations to show that the stationary point of the 
functional (1.108) is given by (1.108). Then use the constraints (1.105), (1.106), 
and (1.107) to eliminate the Lagrange multipliers and hence show that the maximum 
entropy solution is given by the Gaussian (1.109). 


(x) FEY Use the results (1.106) and (1.107) to show that the entropy of the 
univariate Gaussian (1.109) is given by (1.110). 


(x) A strictly convex function is defined as one for which every chord lies above 
the function. Show that this is equivalent to the condition that the second derivative 
of the function be positive. 


(x) Using the definition (1.111) together with the product rule of probability, prove 
the result (1.112). 


(xx) www | Using proof by induction, show that the inequality (1.114) for convex 
functions implies the result (1.115). 


(x* x) Consider two binary variables x and y having the joint distribution given in 
Table 1.3. 


Evaluate the following quantities 


(a) H[z] (c) H[y|z] (e) H[z, y] 
(b) H[y] (d) H[z|y] (f) Ilx, y]. 


Draw a diagram to show the relationship between these various quantities. 
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1.40 (x) By applying Jensen’s inequality (1.115) with f(x) = ln z, show that the arith- 
metic mean of a set of real numbers is never less than their geometrical mean. 


1.41 (x) [QQ Using the sum and product rules of probability, show that the mutual 
information I(x, y) satisfies the relation (1.121). 


> 
(ff 


~ - pl. aa 


Probability 


<=> y 


In Chapter 1, we emphasized the central role played by probability theory in the 
solution of pattern recognition problems. We turn now to an exploration of some 
particular examples of probability distributions and their properties. As well as be- 
ing of great interest in their own right, these distributions can form building blocks 
for more complex models and will be used extensively throughout the book. The 
distributions introduced in this chapter will also serve another important purpose, 
namely to provide us with the opportunity to discuss some key statistical concepts, 
such as Bayesian inference, in the context of simple models before we encounter 
them in more complex situations in later chapters. 

One role for the distributions discussed in this chapter is to model the prob- 
ability distribution p(x) of a random variable x, given a finite set x1,..., Xy of 
observations. This problem is known as density estimation. For the purposes of 
this chapter, we shall assume that the data points are independent and identically 
distributed. It should be emphasized that the problem of density estimation is fun- 
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2.1. 


damentally ill-posed, because there are infinitely many probability distributions that 
could have given rise to the observed finite data set. Indeed, any distribution p(x) 
that is nonzero at each of the data points x,...,xy is a potential candidate. The 
issue of choosing an appropriate distribution relates to the problem of model selec- 
tion that has already been encountered in the context of polynomial curve fitting in 
Chapter 1 and that is a central issue in pattern recognition. 

We begin by considering the binomial and multinomial distributions for discrete 
random variables and the Gaussian distribution for continuous random variables. 
These are specific examples of parametric distributions, so-called because they are 
governed by a small number of adaptive parameters, such as the mean and variance in 
the case of a Gaussian for example. To apply such models to the problem of density 
estimation, we need a procedure for determining suitable values for the parameters, 
given an observed data set. In a frequentist treatment, we choose specific values 
for the parameters by optimizing some criterion, such as the likelihood function. By 
contrast, in a Bayesian treatment we introduce prior distributions over the parameters 
and then use Bayes’ theorem to compute the corresponding posterior distribution 
given the observed data. 

We shall see that an important role is played by conjugate priors, that lead to 
posterior distributions having the same functional form as the prior, and that there- 
fore lead to a greatly simplified Bayesian analysis. For example, the conjugate prior 
for the parameters of the multinomial distribution is called the Dirichlet distribution, 
while the conjugate prior for the mean of a Gaussian is another Gaussian. All of these 
distributions are examples of the exponential family of distributions, which possess 
a number of important properties, and which will be discussed in some detail. 

One limitation of the parametric approach is that it assumes a specific functional 
form for the distribution, which may turn out to be inappropriate for a particular 
application. An alternative approach is given by nonparametric density estimation 
methods in which the form of the distribution typically depends on the size of the data 
set. Such models still contain parameters, but these control the model complexity 
rather than the form of the distribution. We end this chapter by considering three 
nonparametric methods based respectively on histograms, nearest-neighbours, and 
kernels. 


Binary Variables 


We begin by considering a single binary random variable x € {0,1}. For example, 
x might describe the outcome of flipping a coin, with x = 1 representing ‘heads’, 
and x = 0 representing ‘tails’. We can imagine that this is a damaged coin so that 
the probability of landing heads is not necessarily the same as that of landing tails. 
The probability of x = 1 will be denoted by the parameter ju so that 


pia = 1|u) = (2.1) 


Exercise 2.1 
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where 0 < u < 1, from which it follows that p(x = 0|) = 1 — u. The probability 
distribution over x can therefore be written in the form 


Bern(z|u) = p” (1 — p)” (2.2) 
which is known as the Bernoulli distribution. It is easily verified that this distribution 
is normalized and that it has mean and variance given by 
Elze] = pu (2.3) 
varje] = p(l— yp). (2.4) 


Now suppose we have a data set D = {z1,..., £y} of observed values of x. 
We can construct the likelihood function, which is a function of jz, on the assumption 
that the observations are drawn independently from p(z|u), so that 


N 


p(D|u) = [reni = I 


pen (1 — u) *, (2.5) 


In a frequentist setting, we can estimate a value for u by maximizing the likelihood 
function, or equivalently by maximizing the logarithm of the likelihood. In the case 
of the Bernoulli distribution, the log likelihood function is given by 


Sonn £n|u) = 


At this point, it is worth noting that the log likelihood function depends on the N 
observations £n only through their sum )>, £n. This sum provides an example of a 
sufficient statistic for the data under this distribution, and we shall study the impor- 
tant role of sufficient statistics in some detail. If we set the derivative of In p(D|) 


In p(D|w) = ener ces u)}. (2.6) 


n=1 


with respect to u equal to zero, we obtain the maximum likelihood estimator 


Jacob Bernoulli 
1654-1705 


Jacob Bernoulli, also known as 
Jacques or James Bernoulli, was a 
_, Swiss mathematician and was the 
first of many in the Bernoulli family 
to pursue a career in science and 
mathematics. Although compelled 
to study philosophy and theology against his will by 
his parents, he travelled extensively after graduating 
in order to meet with many of the leading scientists of 


UML = 


1 N 
7 5 Tn (2.7) 
n=1 


his time, including Boyle and Hooke in England. When 
he returned to Switzerland, he taught mechanics and 
became Professor of Mathematics at Basel in 1687. 
Unfortunately, rivalry between Jacob and his younger 
brother Johann turned an initially productive collabora- 
tion into a bitter and public dispute. Jacob’s most sig- 
nificant contributions to mathematics appeared in The 
Art of Conjecture published in 1713, eight years after 
his death, which deals with topics in probability the- 
ory including what has become known as the Bernoulli 
distribution. 
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Figure 2.1 


Exercise 2.3 


Histogram plot of the binomial dis- 
tribution (2.9) as a function of m for 
N = 10 and p = 0.25. 


0.3 


which is also known as the sample mean. If we denote the number of observations 
of x = 1 (heads) within this data set by m, then we can write (2.7) in the form 


m= (2.8) 


N 
so that the probability of landing heads is given, in this maximum likelihood frame- 
work, by the fraction of observations of heads in the data set. 

Now suppose we flip a coin, say, 3 times and happen to observe 3 heads. Then 
N = m = 3 and um = 1. In this case, the maximum likelihood result would 
predict that all future observations should give heads. Common sense tells us that 
this is unreasonable, and in fact this is an extreme example of the over-fitting associ- 
ated with maximum likelihood. We shall see shortly how to arrive at more sensible 
conclusions through the introduction of a prior distribution over u. 

We can also work out the distribution of the number m of observations of x = 1, 
given that the data set has size N. This is called the binomial distribution, and 
from (2.5) we see that it is proportional to y™(1 — p)N=™. In order to obtain the 
normalization coefficient we note that out of N coin flips, we have to add up all 
of the possible ways of obtaining m heads, so that the binomial distribution can be 
written 


N 
Bin(m|N, p) = (*) ies las (2.9) 


N N! 
(*) = (N —m)!m! a 


is the number of ways of choosing m objects out of a total of N identical objects. 
Figure 2.1 shows a plot of the binomial distribution for N = 10 and yp = 0.25. 

The mean and variance of the binomial distribution can be found by using the 
result of Exercise 1.10, which shows that for independent events the mean of the 
sum is the sum of the means, and the variance of the sum is the sum of the variances. 
Because m = zı +... + £y, and for each observation the mean and variance are 


where 


Exercise 2.4 


Exercise 2.5 


Exercise 2.6 
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given by (2.3) and (2.4), respectively, we have 


N 
Elm] = 5 mBin(m|N, u) = Nu (2.11) 
m=0 
N 
var|m] = x (m—E[m])* Bin(m|N, yp) = Nul- n). (2.12) 
m=0 


These results can also be proved directly using calculus. 


2.1.1 The beta distribution 


We have seen in (2.8) that the maximum likelihood setting for the parameter ju 
in the Bernoulli distribution, and hence in the binomial distribution, is given by the 
fraction of the observations in the data set having x = 1. As we have already noted, 
this can give severely over-fitted results for small data sets. In order to develop a 
Bayesian treatment for this problem, we need to introduce a prior distribution p(x) 
over the parameter u. Here we consider a form of prior distribution that has a simple 
interpretation as well as some useful analytical properties. To motivate this prior, 
we note that the likelihood function takes the form of the product of factors of the 
form "(1 — y)'~*. If we choose a prior to be proportional to powers of and 
(1 — u), then the posterior distribution, which is proportional to the product of the 
prior and the likelihood function, will have the same functional form as the prior. 
This property is called conjugacy and we will see several examples of it later in this 
chapter. We therefore choose a prior, called the beta distribution, given by 


T(a+b) 
P(a)P(b) 


where T(x) is the gamma function defined by (1.141), and the coefficient in (2.13) 
ensures that the beta distribution is normalized, so that 


Beta(pula, b) = [ima ee) ce (2.13) 


1 
f Beta(yla, b) du = 1. (2.14) 
0 
The mean and variance of the beta distribution are given by 
a 
E = —— 2.15 
[a] Tl (2.15) 


ab 
(a+b) (a+b+1) 


The parameters a and b are often called hyperparameters because they control the 
distribution of the parameter u. Figure 2.2 shows plots of the beta distribution for 
various values of the hyperparameters. 

The posterior distribution of u is now obtained by multiplying the beta prior 
(2.13) by the binomial likelihood function (2.9) and normalizing. Keeping only the 
factors that depend on u, we see that this posterior distribution has the form 


uy (2.17) 


varfu] (2.16) 


plu|m,l, a,b) x pret — 
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H 


Figure 2.2 Plots of the beta distribution Beta(u|a, b) given by (2.13) as a function of „u for various values of the 
hyperparameters a and b. 


where | = N — m, and therefore corresponds to the number of ‘tails’ in the coin 
example. We see that (2.17) has the same functional dependence on ju as the prior 
distribution, reflecting the conjugacy properties of the prior with respect to the like- 
lihood function. Indeed, it is simply another beta distribution, and its normalization 
coefficient can therefore be obtained by comparison with (2.13) to give 


_ Tim + a+ i+ 6) m+a-1 


_ ,,\i+b-1 
T(m +a (l+ b) (hq) ; (2.18) 


p(u|m, l, a, b) 


We see that the effect of observing a data set of m observations of x = 1 and 
l observations of x = 0 has been to increase the value of a by m, and the value of 
b by L, in going from the prior distribution to the posterior distribution. This allows 
us to provide a simple interpretation of the hyperparameters a and b in the prior as 
an effective number of observations of x = 1 and x = 0, respectively. Note that 
a and b need not be integers. Furthermore, the posterior distribution can act as the 
prior if we subsequently observe additional data. To see this, we can imagine taking 
observations one at a time and after each observation updating the current posterior 
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prior 


2 
likelihood function posterior 


1 0 0.5 1 0 0.5 1 
u H 


Figure 2.3 Illustration of one step of sequential Bayesian inference. The prior is given by a beta distribution 
with parameters a = 2, b = 2, and the likelihood function, given by (2.9) with N = m = 1, corresponds to a 
single observation of x = 1, so that the posterior is given by a beta distribution with parameters a = 3, b = 2. 


Section 2.3.5 


distribution by multiplying by the likelihood function for the new observation and 
then normalizing to obtain the new, revised posterior distribution. At each stage, the 
posterior is a beta distribution with some total number of (prior and actual) observed 
values for x = 1 and x = 0 given by the parameters a and b. Incorporation of an 
additional observation of x = 1 simply corresponds to incrementing the value of a 
by 1, whereas for an observation of x = 0 we increment b by 1. Figure 2.3 illustrates 
one step in this process. 

We see that this sequential approach to learning arises naturally when we adopt 
a Bayesian viewpoint. It is independent of the choice of prior and of the likelihood 
function and depends only on the assumption of i.i.d. data. Sequential methods make 
use of observations one at a time, or in small batches, and then discard them before 
the next observations are used. They can be used, for example, in real-time learning 
scenarios where a steady stream of data is arriving, and predictions must be made 
before all of the data is seen. Because they do not require the whole data set to be 
stored or loaded into memory, sequential methods are also useful for large data sets. 
Maximun likelihood methods can also be cast into a sequential framework. 

If our goal is to predict, as best we can, the outcome of the next trial, then we 
must evaluate the predictive distribution of x, given the observed data set D. From 
the sum and product rules of probability, this takes the form 


pæ=1UP)= | p= api) an = f (Ddu = EHD. 2.19) 


Using the result (2.18) for the posterior distribution p(u|D), together with the result 
(2.15) for the mean of the beta distribution, we obtain 


m+a 


Me DY pie 


(2.20) 
which has a simple interpretation as the total fraction of observations (both real ob- 
servations and fictitious prior observations) that correspond to x = 1. Note that in 
the limit of an infinitely large data set m,l — oo the result (2.20) reduces to the 
maximum likelihood result (2.8). As we shall see, it is a very general property that 
the Bayesian and maximum likelihood results will agree in the limit of an infinitely 
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Exercise 2.8 


2.2. 


large data set. For a finite data set, the posterior mean for u always lies between the 
prior mean and the maximum likelihood estimate for u corresponding to the relative 
frequencies of events given by (2.7). 

From Figure 2.2, we see that as the number of observations increases, so the 
posterior distribution becomes more sharply peaked. This can also be seen from 
the result (2.16) for the variance of the beta distribution, in which we see that the 
variance goes to zero for a — oo or b — on. In fact, we might wonder whether it is 
a general property of Bayesian learning that, as we observe more and more data, the 
uncertainty represented by the posterior distribution will steadily decrease. 

To address this, we can take a frequentist view of Bayesian learning and show 
that, on average, such a property does indeed hold. Consider a general Bayesian 
inference problem for a parameter 0 for which we have observed a data set D, de- 
scribed by the joint distribution p(@,D). The following result 


Eo [0] = Ep [Eo[0|D]] (2.21) 


where 


E9(6] = i p(0)0 d0 (2.22) 


i { f 0p(0;D) a0 WD) aD (2.23) 


says that the posterior mean of 0, averaged over the distribution generating the data, 
is equal to the prior mean of @. Similarly, we can show that 


Ep |[Eo[6|D]| 


Ill 


varg[8] = Ep [vare|[O|D]] + varp [Ee[O|D]] . (2.24) 


The term on the left-hand side of (2.24) is the prior variance of 0. On the right- 
hand side, the first term is the average posterior variance of 8, and the second term 
measures the variance in the posterior mean of 0. Because this variance is a positive 
quantity, this result shows that, on average, the posterior variance of 0 is smaller than 
the prior variance. The reduction in variance is greater if the variance in the posterior 
mean is greater. Note, however, that this result only holds on average, and that for a 
particular observed data set it is possible for the posterior variance to be larger than 
the prior variance. 


Multinomial Variables 


Binary variables can be used to describe quantities that can take one of two possible 
values. Often, however, we encounter discrete variables that can take on one of K 
possible mutually exclusive states. Although there are various alternative ways to 
express such variables, we shall see shortly that a particularly convenient represen- 
tation is the 1-of-/ scheme in which the variable is represented by a -dimensional 
vector x in which one of the elements £% equals 1, and all remaining elements equal 


Section 2.4 


Appendix E 


2.2. Multinomial Variables 75 


0. So, for instance if we have a variable that can take K = 6 states and a particular 
observation of the variable happens to correspond to the state where x3 = 1, then x 
will be represented by 

x = (0,0,1,0, 0,0)7. (2.25) 


Note that such vectors satisfy ae zk = 1. If we denote the probability of x = 1 
by the parameter ux, then the distribution of x is given 


p(x|) = Nes pe (2.26) 


where ps = (H1,..., H mys and the parameters ju, are constrained to satisfy uk > 0 
and $`, Hk = 1, because they represent probabilities. The distribution (2.26) can be 
regarded as a generalization of the Bernoulli distribution to more than two outcomes. 
It is easily seen that the distribution is normalized 


K 
3 pom = Soe = 1 (2.27) 
x k=1 


and that 


E[x| py] = Dv x(x = (agin) =p (2.28) 


Now consider a data set D of N independent observations x;,...,x~. The 
corresponding likelihood function takes the form 


vD) = TE TL = 1a = Tt (2.29) 


n=1k=1 


We see that the likelihood function depends on the N data points only through the 
K quantities 


mr = Y on (2.30) 


which represent the number of observations of x = 1. These are called the sufficient 
statistics for this distribution. 

In order to find the maximum likelihood solution for u, we need to maximize 
In p(D|) with respect to yz, taking account of the constraint that the py must sum 
to one. This can be achieved using a Lagrange multiplier \ and maximizing 


K K 
Dmu (Sove—). (2.31) 
k=1 k=1 


Setting the derivative of (2.31) with respect to ux to zero, we obtain 


Hk = —mx/A. (2.32) 
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We can solve for the Lagrange multiplier by substituting (2.32) into the constraint 


X; Hk = 1 to give A = —N. Thus we obtain the maximum likelihood solution in 
the form it 
ML 
aS eee 2.33 
Hk N ( ) 
which is the fraction of the N observations for which x; = 1. 
We can consider the joint distribution of the quantities m4, . . . , mg, conditioned 


on the parameters ys and on the total number N of observations. From (2.29) this 
takes the form 


N K 
Mult(mi, Mm2,..., mg|u, N) = ( ) a” (2.34) 
MyM... MEK Pat 
which is known as the multinomial distribution. The normalization coefficient is the 
number of ways of partitioning N objects into K groups of size mı, ..., mg and is 
given by 
N N! 
= ———. (2.35) 
MM2... MK milm!... mg! 


Note that the variables mę are subject to the constraint 
K 
Som, = N. (2.36) 
k=1 


2.2.1 The Dirichlet distribution 


We now introduce a family of prior distributions for the parameters {up} of 
the multinomial distribution (2.34). By inspection of the form of the multinomial 
distribution, we see that the conjugate prior is given by 


K 
plula) x II per! (2.37) 

k=1 
where 0 < uk < 1 and yx uk = 1. Here a,,...,aK are the parameters of the 
distribution, and œ denotes (a1,...,a es Note that, because of the summation 


constraint, the distribution over the space of the {ux} is confined to a simplex of 
dimensionality K — 1, as illustrated for K = 3 in Figure 2.4. 
The normalized form for this distribution is by 


Dituiei= ETN II wor (2.38) 


which is called the Dirichlet distribution. Here T(x) is the gamma function defined 
by (1.141) while 


K 
Qo = > Mp. (2.39) 
k=1 
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Figure 2.4 The Dirichlet distribution over three variables p1, 42,43 H2 
is confined to a simplex (a bounded linear manifold) of 
the form shown, as a consequence of the constraints 
O<pe< land, ws. =1. 
Hı 


H3 


Plots of the Dirichlet distribution over the simplex, for various settings of the param- 
eters œg, are shown in Figure 2.5. 

Multiplying the prior (2.38) by the likelihood function (2.34), we obtain the 
posterior distribution for the parameters {ux } in the form 


p(u|D, a) xX p(D|e)pl ula) x II ret, 


K 
(2.40) 
k=1 


We see that the posterior distribution again takes the form of a Dirichlet distribution, 
confirming that the Dirichlet is indeed a conjugate prior for the multinomial. This 
allows us to determine the normalization coefficient by comparison with (2.38) so 


that 


P(u|D, a) = 


Dir(p|a@ + m) 


T'(ao +N) 


æk+Mk—1 


where we have denoted m = (mı,.. 


Tr(a + m,)---T(ax + mx) Mk 


(2.41) 
k=1 


m K)”. As for the case of the binomial 


distribution with its beta prior, we can interpret the parameters œx of the Dirichlet 
prior as an effective number of observations of x = 1. 
Note that two-state quantities can either be represented as binary variables and 


Lejeune Dirichlet 
1805-1859 


_ Johann Peter Gustav Lejeune 
Dirichlet was a modest and re- 
served mathematician who made 
contributions in number theory, me- 
chanics, and astronomy, and who 
gave the first rigorous analysis of 
His family originated from Richelet 


Fourier series. 
in Belgium, and the name Lejeune Dirichlet comes 


from ‘le jeune de Richelet’ (the young person from 
Richelet). Dirichlet’s first paper, which was published 
in 1825, brought him instant fame. It concerned Fer- 
mat’s last theorem, which claims that there are no 
positive integer solutions to x” + y” = z” for n > 2. 
Dirichlet gave a partial proof for the case n = 5, which 
was sent to Legendre for review and who in turn com- 
pleted the proof. Later, Dirichlet gave a complete proof 
for n = 14, although a full proof of Fermat's last theo- 
rem for arbitrary n had to wait until the work of Andrew 
Wiles in the closing years of the 20'" century. 


78 2. PROBABILITY DISTRIBUTIONS 


Figure 2.5 Plots of the Dirichlet distribution over three variables, where the two horizontal axes are coordinates 
in the plane of the simplex and the vertical axis corresponds to the value of the density. Here {œx} = 0.1 on the 
left plot, {ax} = 1 in the centre plot, and {œx} = 10 in the right plot. 
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2.3. 


modelled using the binomial distribution (2.9) or as 1-of-2 variables and modelled 
using the multinomial distribution (2.34) with K = 2. 


The Gaussian Distribution 


The Gaussian, also known as the normal distribution, is a widely used model for the 
distribution of continuous variables. In the case of a single variable x, the Gaussian 
distribution can be written in the form 


1 1 
N (z|u, 07) = ear E 4} (2.42) 


where u is the mean and co? is the variance. For a D-dimensional vector x, the 
multivariate Gaussian distribution takes the form 


1 1 1 
N (x|, £) = Gn >? spe exp {-36s =p Se n} (2.43) 


where js is a D-dimensional mean vector, X is a D x D covariance matrix, and |¥]| 
denotes the determinant of X. 

The Gaussian distribution arises in many different contexts and can be motivated 
from a variety of different perspectives. For example, we have already seen that for 
a single real variable, the distribution that maximizes the entropy is the Gaussian. 
This property applies also to the multivariate Gaussian. 

Another situation in which the Gaussian distribution arises is when we consider 
the sum of multiple random variables. The central limit theorem (due to Laplace) 
tells us that, subject to certain mild conditions, the sum of a set of random variables, 
which is of course itself a random variable, has a distribution that becomes increas- 
ingly Gaussian as the number of terms in the sum increases (Walker, 1969). We can 
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Figure 2.6 Histogram plots of the mean of N uniformly distributed numbers for various values of N. We 
observe that as N increases, the distribution tends towards a Gaussian. 


illustrate this by considering N variables xı, . 


.., zy each of which has a uniform 


distribution over the interval [0, 1] and then considering the distribution of the mean 
(xı +--:+2yn)/N. For large N, this distribution tends to a Gaussian, as illustrated 


in Figure 2.6. 


In practice, the convergence to a Gaussian as N increases can be 


very rapid. One consequence of this result is that the binomial distribution (2.9), 
which is a distribution over m defined by the sum of N observations of the random 
binary variable x, will tend to a Gaussian as N — oo (see Figure 2.1 for the case of 


N = 10). 


The Gaussian distribution has many important analytical properties, and we shall 
consider several of these in detail. As a result, this section will be rather more tech- 
nically involved than some of the earlier sections, and will require familiarity with 


Appendix C 


various matrix identities. However, we strongly encourage the reader to become pro- 


ficient in manipulating Gaussian distributions using the techniques presented here as 
this will prove invaluable in understanding the more complex models presented in 


later chapters. 


We begin by considering the geometrical form of the Gaussian distribution. The 


Carl Friedrich Gauss 
1777-1855 


It is said that when Gauss went 
to elementary school at age 7, his 
teacher Buttner, trying to keep the 
class occupied, asked the pupils to 
sum the integers from 1 to 100. To 

-- MJ} the teachers amazement, Gauss 
arrived at the answer in a matter of moments by noting 
that the sum can be represented as 50 pairs (1 + 100, 
2+99, etc.) each of which added to 101, giving the an- 
swer 5,050. It is now believed that the problem which 
was actually set was of the same form but somewhat 
harder in that the sequence had a larger starting value 
and a larger increment. Gauss was a German math- 


ematician and scientist with a reputation for being a 
hard-working perfectionist. One of his many contribu- 
tions was to show that least squares can be derived 
under the assumption of normally distributed errors. 
He also created an early formulation of non-Euclidean 
geometry (a self-consistent geometrical theory that vi- 
olates the axioms of Euclid) but was reluctant to dis- 
cuss it openly for fear that his reputation might suffer 
if it were seen that he believed in such a geometry. 
At one point, Gauss was asked to conduct a geodetic 
survey of the state of Hanover, which led to his for- 
mulation of the normal distribution, now also known 
as the Gaussian. After his death, a study of his di- 
aries revealed that he had discovered several impor- 
tant mathematical results years or even decades be- 
fore they were published by others. 
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functional dependence of the Gaussian on x is through the quadratic form 
A? = (x — p)"Z"(x— y) (2.44) 


which appears in the exponent. The quantity A is called the Mahalanobis distance 
from ps to x and reduces to the Euclidean distance when »& is the identity matrix. The 
Gaussian distribution will be constant on surfaces in x-space for which this quadratic 
form is constant. 

First of all, we note that the matrix & can be taken to be symmetric, without 
loss of generality, because any antisymmetric component would disappear from the 
exponent. Now consider the eigenvector equation for the covariance matrix 


Xu; = Aiu; (2.45) 


where 7 = 1,..., D. Because & is a real, symmetric matrix its eigenvalues will be 
real, and its eigenvectors can be chosen to form an orthonormal set, so that 


where J;;; is the 7, 7 element of the identity matrix and satisfies 


_f 1, ifi=j 
fig = { 0, otherwise. (ay) 


The covariance matrix © can be expressed as an expansion in terms of its eigenvec- 
tors in the form 


D 
D= Aua (2.48) 


P ‘ š n =f 
and similarly the inverse covariance matrix © ~ can be expressed as 


=D ful uT. (2.49) 


Substituting (2.49) into (2.44), the quadratic form becomes 


D F 
A? = A 2.50 
2 (2.50) 
where we have defined 
yi = u; (x — u). (2.51) 


We can interpret {y;} as a new coordinate system defined by the orthonormal vectors 
u; that are shifted and rotated with respect to the original x; coordinates. Forming 
the vector y = (y1,---, yp)”, we have 


y =U(x-p) (2.52) 


Figure 2.7 The red curve shows the ellip- 
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tical surface of constant proba- 
bility density for a Gaussian in 
a two-dimensional space x = 
(a1,22) on which the density 
is exp(—1/2) of its value at 
x = p. The major axes of 
the ellipse are defined by the 
eigenvectors u; of the covari- 
ance matrix, with correspond- 
ing eigenvalues A;. 


Tı 


where U is a matrix whose rows are given by u}. From (2.46) it follows that U is 
an orthogonal matrix, i.e., it satisfies UUT = I, and hence also UTU = I, where I 
is the identity matrix. 

The quadratic form, and hence the Gaussian density, will be constant on surfaces 
for which (2.51) is constant. If all of the eigenvalues À; are positive, then these 
surfaces represent ellipsoids, with their centres at yz and their axes oriented along uj, 
and with scaling factors in the directions of the axes given by N; / ? as illustrated in 
Figure 2.7. 

For the Gaussian distribution to be well defined, it is necessary for all of the 
eigenvalues À; of the covariance matrix to be strictly positive, otherwise the dis- 
tribution cannot be properly normalized. A matrix whose eigenvalues are strictly 
positive is said to be positive definite. In Chapter 12, we will encounter Gaussian 
distributions for which one or more of the eigenvalues are zero, in which case the 
distribution is singular and is confined to a subspace of lower dimensionality. If all 
of the eigenvalues are nonnegative, then the covariance matrix is said to be positive 
semidefinite. 

Now consider the form of the Gaussian distribution in the new coordinate system 
defined by the y;. In going from the x to the y coordinate system, we have a Jacobian 
matrix J with elements given by 


Ox; 
OY; a 


where U;; are the elements of the matrix UT. Using the orthonormality property of 
the matrix U, we see that the square of the determinant of the Jacobian matrix is 


aP = [UT]? = fo" U = [uty] =m =3 254 


and hence |J| = 1. Also, the determinant || of the covariance matrix can be written 
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as the product of its eigenvalues, and hence 
D 
s= (2.55) 
j=1 


Thus in the y; coordinate system, the Gaussian distribution takes the form 


D 


1 2 
ply) = p(x)|J| = | [ Bae *P { > } (2.56) 
F 4 


j=1 


which is the product of D independent univariate Gaussian distributions. The eigen- 
vectors therefore define a new set of shifted and rotated coordinates with respect 
to which the joint probability distribution factorizes into a product of independent 
distributions. The integral of the distribution in the y coordinate system is then 


D oo 2 
1 Y; 
foroa = IL / CMLL exp F dyj=1 (257) 
=v 


where we have used the result (1.48) for the normalization of the univariate Gaussian. 
This confirms that the multivariate Gaussian (2.43) is indeed normalized. 

We now look at the moments of the Gaussian distribution and thereby provide an 
interpretation of the parameters yz and ©. The expectation of x under the Gaussian 
distribution is given by 


s 1 1 1 7 
ia soaps | f-o 'x-m) }xax 
1 1 1 
= Qn? [Ep [ex {apeenal (a+ p) dz (2.58) 


where we have changed variables using z = x — yt. We now note that the exponent 
is an even function of the components of z and, because the integrals over these are 
taken over the range (—oo, oo), the term in z in the factor (z + p) will vanish by 
symmetry. Thus 


Ex] = p (2.59) 


and so we refer to yz as the mean of the Gaussian distribution. 

We now consider second order moments of the Gaussian. In the univariate case, 
we considered the second order moment given by E[z?]. For the multivariate Gaus- 
sian, there are D? second order moments given by E[x;2;|, which we can group 
together to form the matrix E[xx7]. This matrix can be written as 


1 1 1 
es — ate — AT tye T 
[xx] EE fef 5 (x u) © (x n) xx dx 


1 1 1 _ 
apres | op {-327= ‘ah (z+ wa p)" a 
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where again we have changed variables using z = x — yz. Note that the cross-terms 
involving uz” and "z will again vanish by symmetry. The term zp” is constant 
and can be taken outside the integral, which itself is unity because the Gaussian 
distribution is normalized. Consider the term involving zz'. Again, we can make 
use of the eigenvector expansion of the covariance matrix given by (2.45), together 
with the completeness of the set of eigenvectors, to write 


D 
j=1 


T 
j 


1 1 1 Ty-1l T 
arn Er | ef- 5 a) ax dz 


2 
1 1 D D D yy? 
T k 
(2r)P/2 |512 > 9 wu; foo|-¥3 Juna 


where y; = u; z, which gives 


$= j=l 


D 
= ý uu k= (2.61) 


i=1 


where we have made use of the eigenvector equation (2.45), together with the fact 
that the integral on the right-hand side of the middle line vanishes by symmetry 
unless ¿ = 7, and in the final line we have made use of the results (1.50) and (2.55), 
together with (2.48). Thus we have 


EjxxT] = pep? + 5. (2.62) 


For single random variables, we subtracted the mean before taking second mo- 
ments in order to define a variance. Similarly, in the multivariate case it is again 
convenient to subtract off the mean, giving rise to the covariance of a random vector 
x defined by 


cov[x] = E [(x — E[x])(x — E[x])"]. (2.63) 


For the specific case of a Gaussian distribution, we can make use of E[x] = yp, 
together with the result (2.62), to give 


cov|x] = X. (2.64) 


Because the parameter matrix X governs the covariance of x under the Gaussian 
distribution, it is called the covariance matrix. 

Although the Gaussian distribution (2.43) is widely used as a density model, it 
suffers from some significant limitations. Consider the number of free parameters in 
the distribution. A general symmetric covariance matrix © will have D(D + 1)/2 
independent parameters, and there are another D independent parameters in u, giv- 
ing D(D + 3)/2 parameters in total. For large D, the total number of parameters 
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Figure 2.8 Contours of constant zə T2 £2 

probability density for a Gaussian 

distribution in two dimensions in 

which the covariance matrix is (a) of 

general form, (b) diagonal, in which 

the elliptical contours are aligned 

with the coordinate axes, and (c) zı Ly zı 
proportional to the identity matrix, in 

which the contours are concentric (a) (b) (c) 


circles. 
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therefore grows quadratically with D, and the computational task of manipulating 
and inverting large matrices can become prohibitive. One way to address this prob- 
lem is to use restricted forms of the covariance matrix. If we consider covariance 
matrices that are diagonal, so that © = diag(a?), we then have a total of 2D inde- 
pendent parameters in the density model. The corresponding contours of constant 
density are given by axis-aligned ellipsoids. We could further restrict the covariance 
matrix to be proportional to the identity matrix, © = o7I, known as an isotropic co- 
variance, giving D + 1 independent parameters in the model and spherical surfaces 
of constant density. The three possibilities of general, diagonal, and isotropic covari- 
ance matrices are illustrated in Figure 2.8. Unfortunately, whereas such approaches 
limit the number of degrees of freedom in the distribution and make inversion of the 
covariance matrix a much faster operation, they also greatly restrict the form of the 
probability density and limit its ability to capture interesting correlations in the data. 

A further limitation of the Gaussian distribution is that it is intrinsically uni- 
modal (i.e., has a single maximum) and so is unable to provide a good approximation 
to multimodal distributions. Thus the Gaussian distribution can be both too flexible, 
in the sense of having too many parameters, while also being too limited in the range 
of distributions that it can adequately represent. We will see later that the introduc- 
tion of latent variables, also called hidden variables or unobserved variables, allows 
both of these problems to be addressed. In particular, a rich family of multimodal 
distributions is obtained by introducing discrete latent variables leading to mixtures 
of Gaussians, as discussed in Section 2.3.9. Similarly, the introduction of continuous 
latent variables, as described in Chapter 12, leads to models in which the number of 
free parameters can be controlled independently of the dimensionality D of the data 
space while still allowing the model to capture the dominant correlations in the data 
set. Indeed, these two approaches can be combined and further extended to derive 
a very rich set of hierarchical models that can be adapted to a broad range of prac- 
tical applications. For instance, the Gaussian version of the Markov random field, 
which is widely used as a probabilistic model of images, is a Gaussian distribution 
over the joint space of pixel intensities but rendered tractable through the imposition 
of considerable structure reflecting the spatial organization of the pixels. Similarly, 
the linear dynamical system, used to model time series data for applications such 
as tracking, is also a joint Gaussian distribution over a potentially large number of 
observed and latent variables and again is tractable due to the structure imposed on 
the distribution. A powerful framework for expressing the form and properties of 
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such complex distributions is that of probabilistic graphical models, which will form 
the subject of Chapter 8. 


2.3.1 Conditional Gaussian distributions 


An important property of the multivariate Gaussian distribution is that if two 
sets of variables are jointly Gaussian, then the conditional distribution of one set 
conditioned on the other is again Gaussian. Similarly, the marginal distribution of 
either set is also Gaussian. 

Consider first the case of conditional distributions. Suppose x is a D-dimensional 
vector with Gaussian distribution N (x|, ©) and that we partition x into two dis- 
joint subsets x, and x». Without loss of generality, we can take x, to form the first 
M components of x, with x, comprising the remaining D — M components, so that 


Xa 
r= & | (2.65) 


We also define corresponding partitions of the mean vector yz given by 


— [Ha 
p= @ (2.66) 


and of the covariance matrix © given by 
Daa Dab 
y= ; 2.67 
& =) on 


Note that the symmetry ST = X of the covariance matrix implies that Xaa and Nop 
are symmetric, while Ypa = =|; 

In many situations, it will be convenient to work with the inverse of the covari- 
ance matrix 


Aaa (2.68) 


which is known as the precision matrix. In fact, we shall see that some properties 
of Gaussian distributions are most naturally expressed in terms of the covariance, 
whereas others take a simpler form when viewed in terms of the precision. We 
therefore also introduce the partitioned form of the precision matrix 


Naa Nab 
A= ts (2.69) 
corresponding to the partitioning (2.65) of the vector x. Because the inverse of a 
symmetric matrix is also symmetric, we see that Aaa and Aj», are symmetric, while 
AT, = Aba. It should be stressed at this point that, for instance, Aaa is not simply 
given by the inverse of Xaa. In fact, we shall shortly examine the relation between 
the inverse of a partitioned matrix and the inverses of its partitions. 


Let us begin by finding an expression for the conditional distribution p(xq|xp). 
From the product rule of probability, we see that this conditional distribution can be 
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evaluated from the joint distribution p(x) = p(xq,x») simply by fixing x, to the 
observed value and normalizing the resulting expression to obtain a valid probability 
distribution over Xa. Instead of performing this normalization explicitly, we can 
obtain the solution more efficiently by considering the quadratic form in the exponent 
of the Gaussian distribution given by (2.44) and then reinstating the normalization 
coefficient at the end of the calculation. If we make use of the partitioning (2.65), 
(2.66), and (2.69), we obtain 


—F(% — p) E(x- p) = 


1 i 
=3 a= Ha)” Aaa(Xa — Ha) — 5 (Xa — Ha)” Aav(Xb — Hy) 


-E (ep = Hp)" Mala ~ Ha) ~ 5 (6 — Hy)” AC ~ p). (2.70) 
We see that as a function of x,, this is again a quadratic form, and hence the cor- 
responding conditional distribution p(x,|x,) will be Gaussian. Because this distri- 
bution is completely characterized by its mean and its covariance, our goal will be 
to identify expressions for the mean and covariance of p(xq|x») by inspection of 
(2.70). 

This is an example of a rather common operation associated with Gaussian 
distributions, sometimes called ‘completing the square’, in which we are given a 
quadratic form defining the exponent terms in a Gaussian distribution, and we need 
to determine the corresponding mean and covariance. Such problems can be solved 
straightforwardly by noting that the exponent in a general Gaussian distribution 
N (x|, ©) can be written 


1 1 
-3% =p) =" a= u) = -3x Ex +x" ~tu + const (2.71) 


where ‘const’ denotes terms which are independent of x, and we have made use of 
the symmetry of X. Thus if we take our general quadratic form and express it in 
the form given by the right-hand side of (2.71), then we can immediately equate the 
matrix of coefficients entering the second order term in x to the inverse covariance 
matrix ©‘ and the coefficient of the linear term in x to X~! u, from which we can 
obtain u. 

Now let us apply this procedure to the conditional Gaussian distribution p(xa|x») 
for which the quadratic form in the exponent is given by (2.70). We will denote the 
mean and covariance of this distribution by pajp and Xajb, respectively. Consider 
the functional dependence of (2.70) on xq in which x, is regarded as a constant. If 
we pick out all terms that are second order in xa, we have 


1 
= 5a NaaXa (2.72) 
from which we can immediately conclude that the covariance (inverse precision) of 
p(Xa|Xp) is given by 
Se = A aas (2.73) 
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Now consider all of the terms in (2.70) that are linear in x, 
X3 {Aaa Ha — Aav(Xo — Ho) } (2.74) 


where we have used Aba = Aab. From our discussion of the general form (2.71), 
the coefficient of Xa in this expression must equal Da oHalb and hence 


Hab = Zajo {Aaa ha = Nav (Xp = L,)} 
= Ha — Aza Aab(Xb — My) (2.75) 


where we have made use of (2.73). 

The results (2.73) and (2.75) are expressed in terms of the partitioned precision 
matrix of the original joint distribution p(x,, xp). We can also express these results 
in terms of the corresponding partitioned covariance matrix. To do this, we make use 
of the following identity for the inverse of a partitioned matrix 


A B\ M -MBD-! ara 
C D ~\-D-!CM D-!4+D-!CMBD~! j 
where we have defined 
M = (A —- BD C)+. (2.17) 


The quantity M~! is known as the Schur complement of the matrix on the left-hand 
side of (2.76) with respect to the submatrix D. Using the definition 


=a 
aa: Dab Naa Nab 

= 2.78 

i T m z) SR) 


and making use of (2.76), we have 


Naa = (Daa = Sik, Sa) (2.79) 
Nap = —(Laea — Sule Dia) DS ae (2.80) 


From these we obtain the following expressions for the mean and covariance of the 
conditional distribution p(xq|xp) 


Hajo = Hat EX (Xo — H) (2.81) 
Sao = ag Baby Dee (2.82) 


Comparing (2.73) and (2.82), we see that the conditional distribution p(x,|x,) takes 
a simpler form when expressed in terms of the partitioned precision matrix than 
when it is expressed in terms of the partitioned covariance matrix. Note that the 
mean of the conditional distribution p(xq|x»), given by (2.81), is a linear function of 
x, and that the covariance, given by (2.82), is independent of xa. This represents an 
example of a linear-Gaussian model. 
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2.3.2 Marginal Gaussian distributions 


We have seen that if a joint distribution p(xa, xX») is Gaussian, then the condi- 
tional distribution p(x,|x,) will again be Gaussian. Now we turn to a discussion of 
the marginal distribution given by 


2x) = | P(Xa, Xb) dXb (2.83) 


which, as we shall see, is also Gaussian. Once again, our strategy for evaluating this 
distribution efficiently will be to focus on the quadratic form in the exponent of the 
joint distribution and thereby to identify the mean and covariance of the marginal 
distribution p(xa). 

The quadratic form for the joint distribution can be expressed, using the par- 
titioned precision matrix, in the form (2.70). Because our goal is to integrate out 
Xp, this is most easily achieved by first considering the terms involving x, and then 
completing the square in order to facilitate integration. Picking out just those terms 
that involve xp, we have 


1 1 _ = 1 _ 
— 5p Awx tx; m EE — 5 0—Ap m) A(x -A p m)+ 5m" A, m (2.84) 


where we have defined 
m = Abb hp = Ava(Xa = Ha): (2.85) 


We see that the dependence on x, has been cast into the standard quadratic form of a 
Gaussian distribution corresponding to the first term on the right-hand side of (2.84), 
plus a term that does not depend on x, (but that does depend on xa). Thus, when 
we take the exponential of this quadratic form, we see that the integration over x, 
required by (2.83) will take the form 


1 
f exp f -Ze — Ap m) "A (Xo — Ajim)} dxp. (2.86) 


This integration is easily performed by noting that it is the integral over an unnor- 
malized Gaussian, and so the result will be the reciprocal of the normalization co- 
efficient. We know from the form of the normalized Gaussian given by (2.43), that 
this coefficient is independent of the mean and depends only on the determinant of 
the covariance matrix. Thus, by completing the square with respect to x», we can 
integrate out x, and the only term remaining from the contributions on the left-hand 
side of (2.84) that depends on x, is the last term on the right-hand side of (2.84) in 
which m is given by (2.85). Combining this term with the remaining terms from 
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(2.70) that depend on Xa, we obtain 
1 E 
5 [Aobh — Aba (Xa — Ha)]” Agy [Avoti — Aba(Xa — Ha) 
1 
—5%aAaaXa + x7(Naablg + Nabby) + const 


1 
=< — 5% (Aaa = Aab Apy Nia Xe 
+x?) (Aaa — AabAj, Aba) “Hg + const (2.87) 


where ‘const’ denotes quantities independent of x,. Again, by comparison with 
(2.71), we see that the covariance of the marginal distribution of p(x.) is given by 


Da = (Aaa — AwA, Aba) '. (2.88) 
Similarly, the mean is given by 
Eal(Aaa — AAG Aba) Ha = Ha (2.89) 


where we have used (2.88). The covariance in (2.88) is expressed in terms of the 
partitioned precision matrix given by (2.69). We can rewrite this in terms of the 
corresponding partitioning of the covariance matrix given by (2.67), as we did for 
the conditional distribution. These partitioned matrices are related by 


—1 
Naa Nab Daa Dab 
- 2.90 
Oe ie eG a ee) 
Making use of (2.76), we then have 


(Asa -AnA Me) = Dag: (2.91) 


Thus we obtain the intuitively satisfying result that the marginal distribution p(x.) 
has mean and covariance given by 


E[xa] = Ha (2.92) 
cov[xa] = Boa (2.93) 


We see that for a marginal distribution, the mean and covariance are most simply ex- 
pressed in terms of the partitioned covariance matrix, in contrast to the conditional 
distribution for which the partitioned precision matrix gives rise to simpler expres- 
sions. 

Our results for the marginal and conditional distributions of a partitioned Gaus- 
sian are summarized below. 


Partitioned Gaussians 


Given a joint Gaussian distribution N (x| u, ©) with A = 57! and 


_ [Xa _ [Ha 
(e). w(t) m 
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Figure 2.9 The plot on the left shows the contours of a Gaussian distribution p(xza, x») over two variables, and 
the plot on the right shows the marginal distribution p(x.) (blue curve) and the conditional distribution p(xza|z») 
for x, = 0.7 (red curve). 


Dea. Dab Aga Nab 
= A= . 2.95 
& a , = (293) 
Conditional distribution: 
p(xalxo) = N(x|hap Aza) (2.96) 
Hajb = Pa Aza Aab (Xs = Hy). (2.97) 


Marginal distribution: 


p(Xa) SNRs figs Xaa). (2.98) 


We illustrate the idea of conditional and marginal distributions associated with 
a multivariate Gaussian using an example involving two variables in Figure 2.9. 


2.3.3 Bayes’ theorem for Gaussian variables 


In Sections 2.3.1 and 2.3.2, we considered a Gaussian p(x) in which we parti- 
tioned the vector x into two subvectors x = (Xa, X») and then found expressions for 
the conditional distribution p(x,|x;,) and the marginal distribution p(x,). We noted 
that the mean of the conditional distribution p(x,|x,) was a linear function of xp. 
Here we shall suppose that we are given a Gaussian marginal distribution p(x) and a 
Gaussian conditional distribution p(y|x) in which p(y|x) has a mean that is a linear 
function of x, and a covariance which is independent of x. This is an example of 
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a linear Gaussian model (Roweis and Ghahramani, 1999), which we shall study in 

greater generality in Section 8.1.4. We wish to find the marginal distribution p(y) 

and the conditional distribution p(x|y). This is a problem that will arise frequently 

in subsequent chapters, and it will prove convenient to derive the general results here. 
We shall take the marginal and conditional distributions to be 


N (x|p, A~*) (2.99) 
N (y|Ax + b,L*) (2.100) 


p(x) 
P(y|x) 


where u, A, and b are parameters governing the means, and A and L are precision 
matrices. If x has dimensionality M and y has dimensionality D, then the matrix A 
has size D x M. 

First we find an expression for the joint distribution over x and y. To do this, we 


define 
x 
= 2.101 
z G) (2.101) 


and then consider the log of the joint distribution 


Inp(z) = Inp(x) + np(y|x) 
= -ix -p)"A(x- p) 


1 
= Y — Ax — b)"L(y — Ax — b)+ const (2.102) 


where ‘const’ denotes terms independent of x and y. As before, we see that this is a 
quadratic function of the components of z, and hence p(z) is Gaussian distribution. 
To find the precision of this Gaussian, we consider the second order terms in (2.102), 
which can be written as 


i 1 1 1 
SAA + ATLA)x — 5Y Ly + zy LAx + 5X AT Ly 


T 
~ 1[x\Ī/A+ATLA -ATL\ /x\_ lr 


and so the Gaussian distribution over z has precision (inverse covariance) matrix 
given by 
_({(A+ATLA —ATL 
R= ( LA L ) ; (2.104) 


The covariance matrix is found by taking the inverse of the precision, which can be 
done using the matrix inversion formula (2.76) to give 


AT! AHAT ) 


cov[z] = R7* = TE ILAA IAT (2.105) 
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Similarly, we can find the mean of the Gaussian distribution over z by identify- 
ing the linear terms in (2.102), which are given by 


m T 
xTAu—xTATLb +yTLb = a ~ a n , (2.106) 


Using our earlier result (2.71) obtained by completing the square over the quadratic 
form of a multivariate Gaussian, we find that the mean of z is given by 


Elz] = R7! ~ a i . (2.107) 


Making use of (2.105), we then obtain 


E[z] = f i a l (2.108) 


Next we find an expression for the marginal distribution p(y) in which we have 
marginalized over x. Recall that the marginal distribution over a subset of the com- 
ponents of a Gaussian random vector takes a particularly simple form when ex- 
pressed in terms of the partitioned covariance matrix. Specifically, its mean and 
covariance are given by (2.92) and (2.93), respectively. Making use of (2.105) and 
(2.108) we see that the mean and covariance of the marginal distribution p(y) are 
given by 


Ely] = Ap+b (2.109) 
cov[y] = L“! + AAAT. (2.110) 


A special case of this result is when A = I, in which case it reduces to the convolu- 
tion of two Gaussians, for which we see that the mean of the convolution is the sum 
of the mean of the two Gaussians, and the covariance of the convolution is the sum 
of their covariances. 

Finally, we seek an expression for the conditional p(x|y). Recall that the results 
for the conditional distribution are most easily expressed in terms of the partitioned 
precision matrix, using (2.73) and (2.75). Applying these results to (2.105) and 
(2.108) we see that the conditional distribution p(x|y) has mean and covariance 
given by 


E[xly] = (A+ATLA)+{ATL(y -b)+ Ap} (2.111) 
cov[xly] = (A+A7LA)™?. (2.112) 


The evaluation of this conditional can be seen as an example of Bayes’ theorem. 
We can interpret the distribution p(x) as a prior distribution over x. If the variable 
y is observed, then the conditional distribution p(x|y) represents the corresponding 
posterior distribution over x. Having found the marginal and conditional distribu- 
tions, we effectively expressed the joint distribution p(z) = p(x)p(y|x) in the form 
p(xly)p(y). These results are summarized below. 


Appendix C 
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Marginal and Conditional Gaussians 


Given a marginal Gaussian distribution for x and a conditional Gaussian distri- 
bution for y given x in the form 


p(x) = Nella sa”) (2.113) 
p(y|x) = N(y|Ax+b,L7?) (2.114) 


the marginal distribution of y and the conditional distribution of x given y are 
given by 


ply) = N(y|Aut+b,L7'+AA~*A*) (2.115) 
p(xly) = N(x|/E{ATL(y —b) + Ap}, £) (2.116) 

where 
X =(A+A'TLA)?. (2.117) 


2.3.4 Maximum likelihood for the Gaussian 


Given a data set X = (x1,...,xXẸ)" in which the observations {Xn} are as- 
sumed to be drawn independently from a multivariate Gaussian distribution, we can 
estimate the parameters of the distribution by maximum likelihood. The log likeli- 
hood function is given by 


N 
ND N 1 — 
In p(X|p, £) = z In(27) 5 In|] 5 ` (=p) “(ey y). (2.118) 


By simple rearrangement, we see that the likelihood function depends on the data set 
only through the two quantities 


N N 
S Kai De (2.119) 
n=1 n=1 


These are known as the sufficient statistics for the Gaussian distribution. Using 
(C.19), the derivative of the log likelihood with respect to ps is given by 


ð 
Du Inp(X|p, = =e (2.120) 


and setting this derivative to zero, we obtain the solution for the maximum likelihood 
estimate of the mean given by 


1 
Hua. = y bee (2.121) 
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which is the mean of the observed set of data points. The maximization of (2.118) 
with respect to X is rather more involved. The simplest approach is to ignore the 
symmetry constraint and show that the resulting solution is symmetric as required. 
Alternative derivations of this result, which impose the symmetry and positive defi- 
niteness constraints explicitly, can be found in Magnus and Neudecker (1999). The 
result is as expected and takes the form 


N 
XML = Ss — Py) (Xn — My)” (2.122) 


which involves mz because this is the result of a joint maximization with respect 
to u and X. Note that the solution (2.121) for Hyr, does not depend on yp, and so 
we can first evaluate j2),;, and then use this to evaluate Xm. 

If we evaluate the expectations of the maximum likelihood solutions under the 
true distribution, we obtain the following results 


Elin) = p (2.123) 
, N=1 


We see that the expectation of the maximum likelihood estimate for the mean is equal 
to the true mean. However, the maximum likelihood estimate for the covariance has 
an expectation that is less than the true value, and hence it is biased. We can correct 


this bias by defining a different estimator X given by 
aS 1 N 
2= J 2n = pu )(Xn — Hm)”. (2.125) 


Clearly from (2.122) and (2.124), the expectation of ¥ is equal to X. 


2.3.5 Sequential estimation 


Our discussion of the maximum likelihood solution for the parameters of a Gaus- 
sian distribution provides a convenient opportunity to give a more general discussion 
of the topic of sequential estimation for maximum likelihood. Sequential methods 
allow data points to be processed one at a time and then discarded and are important 
for on-line applications, and also where large data sets are involved so that batch 
processing of all data points at once is infeasible. 

Consider the result (2.121) for the maximum likelihood estimator of the mean 
Ly, Which we will denote by pn when it is based on N observations. If we 
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Figure 2.10 Aschematic illustration of two correlated ran- 
dom variables z and 6, together with the 
regression function f(0) given by the con- 
ditional expectation E[z|6]. The Robbins- 
Monro algorithm provides a general sequen- 
tial procedure for finding the root 6* of such 
functions. 


dissect out the contribution from the final data point xy, we obtain 


1 N 
N) _ 
Mun = Wo 
N- 
yer sb» 
1 N-1 gypsy 
= Non N nes 
1 
= pe hoa vp (2.126) 


This result has a nice interpretation, as follows. After observing N — 1 data points 
we have estimated ps by vies We now observe data point xy, and we obtain our 


revised estimate pw ) by moving the old estimate a small amount, proportional to 


1/N, in the direction of the ‘error signal’ (xy — pe ). Note that, as N increases, 


so the contribution from successive data points gets smaller. 

The result (2.126) will clearly give the same answer as the batch result (2.121) 
because the two formulae are equivalent. However, we will not always be able to de- 
rive a sequential algorithm by this route, and so we seek a more general formulation 
of sequential learning, which leads us to the Robbins-Monro algorithm. Consider a 
pair of random variables 0 and z governed by a joint distribution p(z, 0). The con- 
ditional expectation of z given 0 defines a deterministic function f(@) that is given 
by 


f(@) =E[z|6] = [20 dz (2.127) 


and is illustrated schematically in Figure 2.10. Functions defined in this way are 
called regression functions. 

Our goal is to find the root 0* at which f(@*) = 0. If we had a large data set 
of observations of z and 0, then we could model the regression function directly and 
then obtain an estimate of its root. Suppose, however, that we observe values of 
z one at a time and we wish to find a corresponding sequential estimation scheme 
for 6*. The following general procedure for solving such problems was given by 


96 


2. PROBABILITY DISTRIBUTIONS 


Robbins and Monro (1951). We shall assume that the conditional variance of z is 
finite so that 


E [(z — f)? |@] < œ (2.128) 


and we shall also, without loss of generality, consider the case where f (0) > 0 for 
0 > 0* and f(@) < 0 for 0 < 6%, as is the case in Figure 2.10. The Robbins-Monro 
procedure then defines a sequence of successive estimates of the root 0* given by 


9%) = 69ND 4 ay_12(9N-Y) (2.129) 


where z(0?) is an observed value of z when 0 takes the value 0%). The coefficients 
{an } represent a sequence of positive numbers that satisfy the conditions 


lim ay = 0 (2.130) 


Noo 
oo 


X an = œ% (2.131) 
N=1 
X ax < ©. (2.132) 
N=1 


It can then be shown (Robbins and Monro, 1951; Fukunaga, 1990) that the sequence 
of estimates given by (2.129) does indeed converge to the root with probability one. 
Note that the first condition (2.130) ensures that the successive corrections decrease 
in magnitude so that the process can converge to a limiting value. The second con- 
dition (2.131) is required to ensure that the algorithm does not converge short of the 
root, and the third condition (2.132) is needed to ensure that the accumulated noise 
has finite variance and hence does not spoil convergence. 

Now let us consider how a general maximum likelihood problem can be solved 
sequentially using the Robbins-Monro algorithm. By definition, the maximum like- 
lihood solution mz is a stationary point of the log likelihood function and hence 


satisfies x 
o 1 
a0 È Starch 
n=1 


Exchanging the derivative and the summation, and taking the limit N — oo we have 


= 0. (2.133) 


Ou 


19 a 
im W 3 30 In p(£n|0) = Ez È a) (2.134) 


and so we see that finding the maximum likelihood solution corresponds to find- 
ing the root of a regression function. We can therefore apply the Robbins-Monro 
procedure, which now takes the form 


o 
o) — oND + @N—-1 59N) In p(ay|ON-Y), (2.135) 


Figure 2.11 
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In the case of a Gaussian distribution, with 6 Z 

corresponding to the mean ju, the regression 

function illustrated in Figure 2.10 takes the form P(z|H) 
of a straight line, as shown in red. In this 

case, the random variable z corresponds to the 

derivative of the log likelihood function and is 

given by (a — jum) /o, and its expectation that HMI 

defines the regression function is a straight line \ 

given by (u — umL)/o°. The root of the regres- 

sion function corresponds to the maximum like- H 
lihood estimator wm. 


As a specific example, we consider once again the sequential estimation of the 
mean of a Gaussian distribution, in which case the parameter 0) is the estimate 


pen) of the mean of the Gaussian, and the random variable z is given by 


1 
z= In p(z|umL, o°) = 52 le — Lm): (2.136) 


0 
OUML 
Thus the distribution of z is Gaussian with mean u — umu, as illustrated in Fig- 
ure 2.11. Substituting (2.136) into (2.135), we obtain the univariate form of (2.126), 
provided we choose the coefficients ay to have the form ay = o° /N . Note that 
although we have focussed on the case of a single variable, the same technique, 
together with the same restrictions (2.130)-(2.132) on the coefficients ay, apply 
equally to the multivariate case (Blum, 1965). 


2.3.6 Bayesian inference for the Gaussian 


The maximum likelihood framework gave point estimates for the parameters jz 
and X. Now we develop a Bayesian treatment by introducing prior distributions 
over these parameters. Let us begin with a simple example in which we consider a 
single Gaussian random variable x. We shall suppose that the variance g? is known, 
and we consider the task of inferring the mean u given a set of N observations 
X = {xz1,..., £y}. The likelihood function, that is the probability of the observed 
data given u, viewed as a function of ju, is given by 


N N 
1 1 
(Xu) = [| [| pene) = OE ed X (tn - ah (2.137) 
n=1 


n=l 


Again we emphasize that the likelihood function p(X|j/1) is not a probability distri- 
bution over u and is not normalized. 

We see that the likelihood function takes the form of the exponential of a quad- 
ratic form in u. Thus if we choose a prior p(js) given by a Gaussian, it will be a 
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conjugate distribution for this likelihood function because the corresponding poste- 
rior will be a product of two exponentials of quadratic functions of u and hence will 
also be Gaussian. We therefore take our prior distribution to be 


p(t) =N (uluo, 06) (2.138) 
and the posterior distribution is given by 
P(u|X) x p(X] H)p(H). (2.139) 


Simple manipulation involving completing the square in the exponent shows that the 
posterior distribution is given by 


pX) =N (ulun, on) (2.140) 
where 
o? No} 
= 2.141 
Ln Notte Ho Notte? UML ( ) 
1 1 N 
z = ot (2.142) 
on o 


in which umu is the maximum likelihood solution for u given by the sample mean 


1 N 
m= >, Tn. (2.143) 


It is worth spending a moment studying the form of the posterior mean and 
variance. First of all, we note that the mean of the posterior distribution given by 
(2.141) is a compromise between the prior mean jig and the maximum likelihood 
solution jw. If the number of observed data points N = 0, then (2.141) reduces 
to the prior mean as expected. For N — oo, the posterior mean is given by the 
maximum likelihood solution. Similarly, consider the result (2.142) for the variance 
of the posterior distribution. We see that this is most naturally expressed in terms 
of the inverse variance, which is called the precision. Furthermore, the precisions 
are additive, so that the precision of the posterior is given by the precision of the 
prior plus one contribution of the data precision from each of the observed data 
points. As we increase the number of observed data points, the precision steadily 
increases, corresponding to a posterior distribution with steadily decreasing variance. 
With no observed data points, we have the prior variance, whereas if the number of 
data points V — oo, the variance of, goes to zero and the posterior distribution 
becomes infinitely peaked around the maximum likelihood solution. We therefore 
see that the maximum likelihood result of a point estimate for u given by (2.143) is 
recovered precisely from the Bayesian formalism in the limit of an infinite number 
of observations. Note also that for finite N, if we take the limit of — oo in which the 
prior has infinite variance then the posterior mean (2.141) reduces to the maximum 
likelihood result, while from (2.142) the posterior variance is given by oå = 07/N. 
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Illustration of Bayesian inference for 
the mean yw of a Gaussian distri- 
bution, in which the variance is as- 
sumed to be known. The curves 
show the prior distribution over pu 
(the curve labelled N = 0), which 
in this case is itself Gaussian, along 
with the posterior distribution given 
by (2.140) for increasing numbers N 
of data points. The data points are 
generated from a Gaussian of mean 
0.8 and variance 0.1, and the prior is 
chosen to have mean 0. In both the 
prior and the likelihood function, the 
variance is set to the true value. 


We illustrate our analysis of Bayesian inference for the mean of a Gaussian 
distribution in Figure 2.12. The generalization of this result to the case of a D- 
dimensional Gaussian random variable x with known covariance and unknown mean 
is straightforward. 

We have already seen how the maximum likelihood expression for the mean of 
a Gaussian can be re-cast as a sequential update formula in which the mean after 
observing N data points was expressed in terms of the mean after observing N — 1 
data points together with the contribution from data point xy. In fact, the Bayesian 
paradigm leads very naturally to a sequential view of the inference problem. To see 
this in the context of the inference of the mean of a Gaussian, we write the posterior 
distribution with the contribution from the final data point xy separated out so that 


N-1 
p(u|D) x |plu) | [| pen|e)| pxu). (2.144) 


The term in square brackets is (up to a normalization coefficient) just the posterior 
distribution after observing NV — 1 data points. We see that this can be viewed as 
a prior distribution, which is combined using Bayes’ theorem with the likelihood 
function associated with data point xy to arrive at the posterior distribution after 
observing N data points. This sequential view of Bayesian inference is very general 
and applies to any problem in which the observed data are assumed to be independent 
and identically distributed. 

So far, we have assumed that the variance of the Gaussian distribution over the 
data is known and our goal is to infer the mean. Now let us suppose that the mean 
is known and we wish to infer the variance. Again, our calculations will be greatly 
simplified if we choose a conjugate form for the prior distribution. It turns out to be 
most convenient to work with the precision \ = 1/c7. The likelihood function for A 
takes the form 


N N 
= a N/2 ^ =f 
p(X|A) = [l Nala ) œx A“ 4 exp 5 > len oe (2.145) 
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Figure 2.13 Plot of the gamma distribution Gam(A|a, b) defined by (2.146) for various values of the parameters 


a and b. 
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The corresponding conjugate prior should therefore be proportional to the product 
of a power of À and the exponential of a linear function of A. This corresponds to 
the gamma distribution which is defined by 


1 
D(a) 
Here T (a) is the gamma function that is defined by (1.141) and that ensures that 
(2.146) is correctly normalized. The gamma distribution has a finite integral if a > 0, 


and the distribution itself is finite if a > 1. It is plotted, for various values of a and 
b, in Figure 2.13. The mean and variance of the gamma distribution are given by 


Gam(A|a, b) = b*\°—! exp(—bd). (2.146) 


EIA) = (2.147) 


var[A] = (2.148) 


Blacie 


Consider a prior distribution Gam(A|ao, bo). If we multiply by the likelihood 
function (2.145), then we obtain a posterior distribution 


N 
À 
ao—1 3 N/2 py_a oer: 
p(A|X) œ A7 A oof boà 5 > (en nh (2.149) 


which we recognize as a gamma distribution of the form Gam(A|ay, by ) where 


N 
an = aot D (2.150) 
N 
1 N 
by = bots > (2n — u)? = bo + IML (2.151) 


where oĉ; is the maximum likelihood estimator of the variance. Note that in (2.149) 
there is no need to keep track of the normalization constants in the prior and the 
likelihood function because, if required, the correct coefficient can be found at the 
end using the normalized form (2.146) for the gamma distribution. 
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From (2.150), we see that the effect of observing N data points is to increase 
the value of the coefficient a by N/2. Thus we can interpret the parameter ao in 
the prior in terms of 2a ‘effective’ prior observations. Similarly, from (2.151) we 
see that the N data points contribute Noj,,,/2 to the parameter b, where oX;;, is 
the variance, and so we can interpret the parameter bp in the prior as arising from 
the 2a9 ‘effective’ prior observations having variance 2b9/(2a9) = bo/ao. Recall 
that we made an analogous interpretation for the Dirichlet prior. These distributions 
are examples of the exponential family, and we shall see that the interpretation of 
a conjugate prior in terms of effective fictitious data points is a general one for the 
exponential family of distributions. 

Instead of working with the precision, we can consider the variance itself. The 
conjugate prior in this case is called the inverse gamma distribution, although we 
shall not discuss this further because we will find it more convenient to work with 
the precision. 

Now suppose that both the mean and the precision are unknown. To find a 
conjugate prior, we consider the dependence of the likelihood function on u and A 


p(X|u, A) = Il (>) = exp {-Zler - we} 


n=1 


N N N 
x |A! exp A exp < àu > Bin a > x? $. (2.152) 
2 D n 


We now wish to identify a prior distribution p(j:, A) that has the same functional 
dependence on p and A as the likelihood function and that should therefore take the 
form 


2 B 
plu, A) x j exp (5) exp {cAu — dà} 


= opf -Au ejay} NF? exp {- (a- 5) a} (2.153) 


where c, d, and 8 are constants. Since we can always write p(w, A) = p(ulà)p(à), 
we can find p(u|à) and p(A) by inspection. In particular, we see that p(ju|X) is a 
Gaussian whose precision is a linear function of À and that p(A) is a gamma distri- 
bution, so that the normalized prior takes the form 


P(H, A) = N (ulļuo, (A) *)Gam(Ala, b) (2.154) 


where we have defined new constants given by uo = c/3,a = 1+ 8/2, b = 
d—c?/2(. The distribution (2.154) is called the normal-gamma or Gaussian-gamma 
distribution and is plotted in Figure 2.14. Note that this is not simply the product 
of an independent Gaussian prior over u and a gamma prior over À, because the 
precision of u is a linear function of A. Even if we chose a prior in which u and A 
were independent, the posterior distribution would exhibit a coupling between the 
precision of u and the value of A. 


102 2. PROBABILITY DISTRIBUTIONS 


Figure 2.14 Contour plot of the normal-gamma 
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distribution (2.154) for parameter 
values po = 0, 8 = 2, a = 5 and 
b=6. 


In the case of the multivariate Gaussian distribution M (x| H, A`?) for a D- 
dimensional variable x, the conjugate prior distribution for the mean jz, assuming 
the precision is known, is again a Gaussian. For known mean and unknown precision 
matrix A, the conjugate prior is the Wishart distribution given by 


W(A|W, v) = BIA|Y-P-Y?? exp (- bwa) (2.155) 


where v is called the number of degrees of freedom of the distribution, W is a D x D 
scale matrix, and Tr(-) denotes the trace. The normalization constant B is given by 


D ; 
B(W,v) = [wW] Ga qo | T (==) . (2.156) 


4=] 


Again, it is also possible to define a conjugate prior over the covariance matrix itself, 
rather than over the precision matrix, which leads to the inverse Wishart distribu- 
tion, although we shall not discuss this further. If both the mean and the precision 
are unknown, then, following a similar line of reasoning to the univariate case, the 
conjugate prior is given by 


plu, Almo, 8, W, v) = N (ulmo, (8A) W(AIW, v) (2.157) 
which is known as the normal-Wishart or Gaussian-Wishart distribution. 


2.3.7 Student’s t-distribution 


We have seen that the conjugate prior for the precision of a Gaussian is given 
by a gamma distribution. If we have a univariate Gaussian V(a|,7~') together 
with a Gamma prior Gam(r|a, b) and we integrate out the precision, we obtain the 
marginal distribution of x in the form 


Figure 2.15 Plot of Student's t-distribution (2.159) 
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for u = 0 and à = 1 for various values 0:3 


of v. The limit v — oo corresponds 
to a Gaussian distribution with mean 
u and precision A. 


p= CO 


plz|u,a,b) = I N (x|u, r +)Gam(r]a, b) dr (2.158) 
0 


oo bte(—or) ra] r \ 1/2 T 7 
f Tr(a) = exp {=(= u) } 


Fa o b- =] M iie +1/2) 


where we have made the change of variable z = T[b + (x — u)? /2]. By convention 
we define new parameters given by v = 2a and À = a/b, in terms of which the 
distribution p(x|u, a, b) takes the form 


1/2 27 —v/2—1/2 
TTAN a ane 2 ( >) h J M (2.159) 


which is known as Student’s t-distribution. The parameter À is sometimes called the 
precision of the t-distribution, even though it is not in general equal to the inverse 
of the variance. The parameter v is called the degrees of freedom, and its effect is 
illustrated in Figure 2.15. For the particular case of v = 1, the t-distribution reduces 
to the Cauchy distribution, while in the limit v — oo the t-distribution St(z|, A, v) 
becomes a Gaussian M (x| u, A+) with mean pu and precision À. 

From (2.158), we see that Student’s t-distribution is obtained by adding up an 
infinite number of Gaussian distributions having the same mean but different preci- 
sions. This can be interpreted as an infinite mixture of Gaussians (Gaussian mixtures 
will be discussed in detail in Section 2.3.9. The result is a distribution that in gen- 
eral has longer ‘tails’ than a Gaussian, as was seen in Figure 2.15. This gives the t- 
distribution an important property called robustness, which means that it is much less 
sensitive than the Gaussian to the presence of a few data points which are outliers. 
The robustness of the t-distribution is illustrated in Figure 2.16, which compares the 
maximum likelihood solutions for a Gaussian and a t-distribution. Note that the max- 
imum likelihood solution for the t-distribution can be found using the expectation- 
maximization (EM) algorithm. Here we see that the effect of a small number of 
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Figure 2.16 Illustration of the robustness of Student’s t-distribution compared to a Gaussian. (a) Histogram 
distribution of 30 data points drawn from a Gaussian distribution, together with the maximum likelihood fit ob- 
tained from a t-distribution (red curve) and a Gaussian (green curve, largely hidden by the red curve). Because 
the t-distribution contains the Gaussian as a special case it gives almost the same solution as the Gaussian. 
(b) The same data set but with three additional outlying data points showing how the Gaussian (green curve) is 
strongly distorted by the outliers, whereas the t-distribution (red curve) is relatively unaffected. 
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outliers is much less significant for the t-distribution than for the Gaussian. Outliers 
can arise in practical applications either because the process that generates the data 
corresponds to a distribution having a heavy tail or simply through mislabelled data. 
Robustness is also an important property for regression problems. Unsurprisingly, 
the least squares approach to regression does not exhibit robustness, because it cor- 
responds to maximum likelihood under a (conditional) Gaussian distribution. By 
basing a regression model on a heavy-tailed distribution such as a t-distribution, we 
obtain a more robust model. 

If we go back to (2.158) and substitute the alternative parameters v = 2a, À = 
a/b, and n = Tb/a, we see that the t-distribution can be written in the form 


St(z|u, A, v) a N (xu, (nà) +) Gam(n|v/2,v/2) dn. (2.160) 
0 


We can then generalize this to a multivariate Gaussian N (x| u, A) to obtain the cor- 
responding multivariate Student’s t-distribution in the form 


St(xlu, Av) = f N (x|p, (nA)~')Gam(n\v/2, v/2) dn. (2.161) 
0 


Using the same technique as for the univariate case, we can evaluate this integral to 
give 
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St(x|u, A, v) = 


1/2 27 —D/2—v/2 
T(D/2+v/2) |A| | | A (2.162) 


T(v/2) (nv)? 


V 


where D is the dimensionality of x, and A? is the squared Mahalanobis distance 
defined by 
A? = (x — p)™A (x — n). (2.163) 


This is the multivariate form of Student’s t-distribution and satisfies the following 
properties 


Ex] = p, if v>l (2.164) 
cove] = 7 7 3) a if v>2 (2.165) 
mode[x] = p (2.166) 


with corresponding results for the univariate case. 


2.3.8 Periodic variables 


Although Gaussian distributions are of great practical significance, both in their 
own right and as building blocks for more complex probabilistic models, there are 
situations in which they are inappropriate as density models for continuous vari- 
ables. One important case, which arises in practical applications, is that of periodic 
variables. 

An example of a periodic variable would be the wind direction at a particular 
geographical location. We might, for instance, measure values of wind direction on a 
number of days and wish to summarize this using a parametric distribution. Another 
example is calendar time, where we may be interested in modelling quantities that 
are believed to be periodic over 24 hours or over an annual cycle. Such quantities 
can conveniently be represented using an angular (polar) coordinate 0 < 0 < 27. 

We might be tempted to treat periodic variables by choosing some direction 
as the origin and then applying a conventional distribution such as the Gaussian. 
Such an approach, however, would give results that were strongly dependent on the 
arbitrary choice of origin. Suppose, for instance, that we have two observations at 
0, = 1° and 03 = 359°, and we model them using a standard univariate Gaussian 
distribution. If we choose the origin at 0°, then the sample mean of this data set 
will be 180° with standard deviation 179°, whereas if we choose the origin at 180°, 
then the mean will be 0° and the standard deviation will be 1°. We clearly need to 
develop a special approach for the treatment of periodic variables. 

Let us consider the problem of evaluating the mean of a set of observations 
D = {6,,...,0y} of a periodic variable. From now on, we shall assume that 6 is 
measured in radians. We have already seen that the simple average (0, +---+0y)/N 
will be strongly coordinate dependent. To find an invariant measure of the mean, we 
note that the observations can be viewed as points on the unit circle and can therefore 
be described instead by two-dimensional unit vectors x;,...,X, where ||x,,|| = 1 
for n = 1,..., N, as illustrated in Figure 2.17. We can average the vectors {Xn } 
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Figure 2.17 


Illustration of the representation of val- 
ues 0, of a periodic variable as two- 


T2 
: r ie ; X4 
dimensional vectors x» living on the unit 
circle. Also shown is the average x of 
those vectors. 
X2 


X3 


Tı 
x1 


instead to give 
LS 
x= a (2.167) 
n=1 


and then find the corresponding angle 0 of this average. Clearly, this definition will 
ensure that the location of the mean is independent of the origin of the angular coor- 
dinate. Note that x will typically lie inside the unit circle. The Cartesian coordinates 
of the observations are given by Xn = (cos 0,,,sin ôn), and we can write the Carte- 
sian coordinates of the sample mean in the form X = (F cos 0,7 sin @). Substituting 
into (2.167) and equating the xı and x2 components then gives 


ie i 
T cos = TA i Tsin = sA tife (2.168) 
Taking the ratio, and using the identity tan 0 = sin 6/ cos 0, we can solve for 0 to 
give 
= sin ôn 
J = tan! jer}. (2.169) 


Shortly, we shall see how this result arises naturally as the maximum likelihood 
estimator for an appropriately defined distribution over a periodic variable. 

We now consider a periodic generalization of the Gaussian called the von Mises 
distribution. Here we shall limit our attention to univariate distributions, although 
periodic distributions can also be found over hyperspheres of arbitrary dimension. 
For an extensive discussion of periodic distributions, see Mardia and Jupp (2000). 

By convention, we will consider distributions p(@) that have period 27. Any 
probability density p(@) defined over 6 must not only be nonnegative and integrate 
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Figure 2.18 The von Mises distribution can be derived by considering 
a two-dimensional Gaussian of the form (2.173), whose 
density contours are shown in blue and conditioning on 
the unit circle shown in red. 


to one, but it must also be periodic. Thus p(@) must satisfy the three conditions 


po) > 0 (2.170) 
2m 
I p(80)dð = 1 (2.171) 
0 
p(O@+2r) = p8). (2.172) 


From (2.172), it follows that p(@ + M2r) = p(@) for any integer M. 

We can easily obtain a Gaussian-like distribution that satisfies these three prop- 
erties as follows. Consider a Gaussian distribution over two variables x = (x1, £2) 
having mean yt = (u1, 42) and a covariance matrix © = o7I where I is the 2 x 2 
identity matrix, so that 


(2.173) 


1 (xı = p)? + (£2 — be)” 
Plata) = Dro? P { 20? 


The contours of constant p(x) are circles, as illustrated in Figure 2.18. Now suppose 
we consider the value of this distribution along a circle of fixed radius. Then by con- 
struction this distribution will be periodic, although it will not be normalized. We can 
determine the form of this distribution by transforming from Cartesian coordinates 
(£1, £2) to polar coordinates (r, 0) so that 


zı = r cos ð, Tt =r sinb. (2.174) 
We also map the mean p into polar coordinates by writing 
Hı = To COS bo, po = Tro sin 0o. (2.175) 


Next we substitute these transformations into the two-dimensional Gaussian distribu- 
tion (2.173), and then condition on the unit circle r = 1, noting that we are interested 
only in the dependence on @. Focussing on the exponent in the Gaussian distribution 
we have 


1 
— 552 {(r cos 6 — ro cos ĝo)? + (r sin 0 — ro sin 9)?} 
a 


1 
= oe {1+ rg — 2ro cos 8 cos fo — 2ro sin A sin 6o } 
g 


= = cos(@ — 0o) + const (2.176) 
o 
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Figure 2.19 The von Mises distribution plotted for two different parameter values, shown as a Cartesian plot 
on the left and as the corresponding polar plot on the right. 
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Exercise 2.52 


where ‘const’ denotes terms independent of 0, and we have made use of the following 
trigonometrical identities 


cos? A+sin? A = 1 (2.177) 
cos Acos B +sin Asin B = cos(A-— B). (2.178) 


If we now define m = ro/o”, we obtain our final expression for the distribution of 
p(0) along the unit circle r = 1 in the form 


1 

p(9|00,m) = San) exp {m cos(0 — 0) } (2.179) 
which is called the von Mises distribution, or the circular normal. Here the param- 
eter ĝo corresponds to the mean of the distribution, while m, which is known as 
the concentration parameter, is analogous to the inverse variance (precision) for the 
Gaussian. The normalization coefficient in (2.179) is expressed in terms of Io(m), 
which is the zeroth-order Bessel function of the first kind (Abramowitz and Stegun, 
1965) and is defined by 


1 20 
hm) = F exp {mcos 6} dé. (2.180) 
0 


For large m, the distribution becomes approximately Gaussian. The von Mises dis- 
tribution is plotted in Figure 2.19, and the function Jo(m) is plotted in Figure 2.20. 


Now consider the maximum likelihood estimators for the parameters 69 and m 
for the von Mises distribution. The log likelihood function is given by 


N 
In p(D|%,m) = -N In(2r) — N In (m) +m} © cos(n — 8o). (2.181) 
n=1 
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Figure 2.20 Plot of the Bessel function Jo(m) defined by (2.180), together with the function A(m) defined by 


(2.186). 
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Setting the derivative with respect to 0o equal to zero gives 


N 
X sin(ên — 6) = 0. (2.182) 
n=1 
To solve for 0o, we make use of the trigonometric identity 
sin(A — B) = cos Bsin A — cos Asin B (2.183) 
from which we obtain i 
ML = tan`! { ae (2.184) 


which we recognize as the result (2.169) obtained earlier for the mean of the obser- 
vations viewed in a two-dimensional Cartesian space. 

Similarly, maximizing (2.181) with respect to m, and making use of (m) = 
I,(m) (Abramowitz and Stegun, 1965), we have 


N 
A(m) = ~ X cos(On, — 0)™) (2.185) 
n=1 


where we have substituted for the maximum likelihood solution for 6}™ (recalling 
that we are performing a joint optimization over 0 and m), and we have defined 


E I,(m) 
Io(m) i 


The function A(m) is plotted in Figure 2.20. Making use of the trigonometric iden- 
tity (2.178), we can write (2.185) in the form 


N N 
1 1 
A(mu) = (4 ` cnt) cos OME — (> ` snd) sin 6ð™. (2.187) 
n=1 


n=l 


(2.186) 


A(m) 
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Figure 2.21 


bility density. 
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Plots of the ‘old faith- 
ful’ data in which the blue curves 
show contours of constant proba- 
On the left is a 
single Gaussian distribution which 80 
has been fitted to the data us- 
ing maximum likelihood. Note that 
this distribution fails to capture the 
two clumps in the data and indeed 
places much of its probability mass 
in the central region between the 
clumps where the data are relatively 40 
sparse. On the right the distribution 


100 100 


60 


is given by a linear combination of 
two Gaussians which has been fitted 
to the data by maximum likelinood 
using techniques discussed Chap- 
ter 9, and which gives a better rep- 
resentation of the data. 


Appendix A 


The right-hand side of (2.187) is easily evaluated, and the function A(m) can be 
inverted numerically. 

For completeness, we mention briefly some alternative techniques for the con- 
struction of periodic distributions. The simplest approach is to use a histogram of 
observations in which the angular coordinate is divided into fixed bins. This has the 
virtue of simplicity and flexibility but also suffers from significant limitations, as we 
shall see when we discuss histogram methods in more detail in Section 2.5. Another 
approach starts, like the von Mises distribution, from a Gaussian distribution over a 
Euclidean space but now marginalizes onto the unit circle rather than conditioning 
(Mardia and Jupp, 2000). However, this leads to more complex forms of distribution 
and will not be discussed further. Finally, any valid distribution over the real axis 
(such as a Gaussian) can be turned into a periodic distribution by mapping succes- 
sive intervals of width 27 onto the periodic variable (0,27), which corresponds to 
‘wrapping’ the real axis around unit circle. Again, the resulting distribution is more 
complex to handle than the von Mises distribution. 

One limitation of the von Mises distribution is that it is unimodal. By forming 
mixtures of von Mises distributions, we obtain a flexible framework for modelling 
periodic variables that can handle multimodality. For an example of a machine learn- 
ing application that makes use of von Mises distributions, see Lawrence et al. (2002), 
and for extensions to modelling conditional densities for regression problems, see 
Bishop and Nabney (1996). 


2.3.9 Mixtures of Gaussians 


While the Gaussian distribution has some important analytical properties, it suf- 
fers from significant limitations when it comes to modelling real data sets. Consider 
the example shown in Figure 2.21. This is known as the ‘Old Faithful’ data set, 
and comprises 272 measurements of the eruption of the Old Faithful geyser at Yel- 
lowstone National Park in the USA. Each measurement comprises the duration of 
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Figure 2.22 Example of a Gaussian mixture distribution p(x) 


Section 9.3.3 


in one dimension showing three Gaussians 
(each scaled by a coefficient) in blue and 
their sum in red. 


the eruption in minutes (horizontal axis) and the time in minutes to the next erup- 
tion (vertical axis). We see that the data set forms two dominant clumps, and that 
a simple Gaussian distribution is unable to capture this structure, whereas a linear 
superposition of two Gaussians gives a better characterization of the data set. 

Such superpositions, formed by taking linear combinations of more basic dis- 
tributions such as Gaussians, can be formulated as probabilistic models known as 
mixture distributions (McLachlan and Basford, 1988; McLachlan and Peel, 2000). 
In Figure 2.22 we see that a linear combination of Gaussians can give rise to very 
complex densities. By using a sufficient number of Gaussians, and by adjusting their 
means and covariances as well as the coefficients in the linear combination, almost 
any continuous density can be approximated to arbitrary accuracy. 

We therefore consider a superposition of K Gaussian densities of the form 


K 
p(x) = So mN (x| My, Zx) (2.188) 


k=1 


which is called a mixture of Gaussians. Each Gaussian density N (x| up, 2x) is 
called a component of the mixture and has its own mean py, and covariance Nx. 
Contour and surface plots for a Gaussian mixture having 3 components are shown in 
Figure 2.23. 

In this section we shall consider Gaussian components to illustrate the frame- 
work of mixture models. More generally, mixture models can comprise linear com- 
binations of other distributions. For instance, in Section 9.3.3 we shall consider 
mixtures of Bernoulli distributions as an example of a mixture model for discrete 
variables. 

The parameters my in (2.188) are called mixing coefficients. If we integrate both 
sides of (2.188) with respect to x, and note that both p(x) and the individual Gaussian 
components are normalized, we obtain 


K 
Som =i (2.189) 
k=1 


Also, the requirement that p(x) > 0, together with N (x|up, 4) > 0, implies 
Tk 2 0 for all k. Combining this with the condition (2.189) we obtain 


O<m <1. (2.190) 
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Figure 2.23 Illustration of a mixture of 3 Gaussians in a two-dimensional space. (a) Contours of constant 
density for each of the mixture components, in which the 3 components are denoted red, blue and green, and 
the values of the mixing coefficients are shown below each component. (b) Contours of the marginal probability 
density p(x) of the mixture distribution. (c) A surface plot of the distribution p(x). 


We therefore see that the mixing coefficients satisfy the requirements to be probabil- 
ities. 
From the sum and product rules, the marginal density is given by 


K 
p(x) = X_ p(k)p(x|k) (2.191) 
k=1 


which is equivalent to (2.188) in which we can view 7, = p(k) as the prior prob- 
ability of picking the k*® component, and the density N (x|up, Sx) = p(x|k) as 
the probability of x conditioned on k. As we shall see in later chapters, an impor- 
tant role is played by the posterior probabilities p(k|x), which are also known as 
responsibilities. From Bayes’ theorem these are given by 


p(k|x) 
p(k)p(x|k) 

X ppl) 
TN (xl My, Dr) 


> TUN (x| M1, X) 


We shall discuss the probabilistic interpretation of the mixture distribution in greater 


Yn (X) 


(2.192) 


detail in Chapter 9. 
The form of the Gaussian mixture distribution is governed by the parameters 7, 
p and ©, where we have used the notation m = {m,...,7K}, w= {M),---, UK} 


and X = {%j,...~%}. One way to set the values of these parameters is to use 
maximum likelihood. From (2.188) the log of the likelihood function is given by 


N K 
In p(X|z, p, £) = Xoin [S neuzo (2.193) 
k=1 


n=1 


2.4. 
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where X = {x,,...,xy}. We immediately see that the situation is now much 
more complex than with a single Gaussian, due to the presence of the summation 
over k inside the logarithm. As a result, the maximum likelihood solution for the 
parameters no longer has a closed-form analytical solution. One approach to maxi- 
mizing the likelihood function is to use iterative numerical optimization techniques 
(Fletcher, 1987; Nocedal and Wright, 1999; Bishop and Nabney, 2008). Alterna- 
tively we can employ a powerful framework called expectation maximization, which 
will be discussed at length in Chapter 9. 


The Exponential Family 


The probability distributions that we have studied so far in this chapter (with the 
exception of the Gaussian mixture) are specific examples of a broad class of distri- 
butions called the exponential family (Duda and Hart, 1973; Bernardo and Smith, 
1994). Members of the exponential family have many important properties in com- 
mon, and it is illuminating to discuss these properties in some generality. 

The exponential family of distributions over x, given parameters n, is defined to 
be the set of distributions of the form 


p(x|n) = h(x)g(n) exp {n* u(x) } (2.194) 


where x may be scalar or vector, and may be discrete or continuous. Here 77 are 
called the natural parameters of the distribution, and u(x) is some function of x. 
The function g(7) can be interpreted as the coefficient that ensures that the distribu- 
tion is normalized and therefore satisfies 


g(n) / h(x) exp {n'u(x) } dx = 1 (2.195) 


where the integration is replaced by summation if x is a discrete variable. 

We begin by taking some examples of the distributions introduced earlier in 
the chapter and showing that they are indeed members of the exponential family. 
Consider first the Bernoulli distribution 


p(z|u) = Bern(z|u) = p? (1 — p)”. (2.196) 


Expressing the right-hand side as the exponential of the logarithm, we have 


p(æ|u) = exp{zlnu+ (1-— x)In(1 — u)} 


= (1—p)exp {in (4) e} (2.197) 


Comparison with (2.194) allows us to identify 


n=In (4) (2.198) 
l-yp 
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which we can solve for ju to give u = o(7), where 


1 


= eee eT, (2.199) 


a(n) 


is called the logistic sigmoid function. Thus we can write the Bernoulli distribution 
using the standard representation (2.194) in the form 


p(z|n) = o(—n) exp(nz) (2.200) 


where we have used 1 — o (n) = a(—nņ), which is easily proved from (2.199). Com- 
parison with (2.194) shows that 


u(t) = z (2.201) 
A(z) = 1 (2.202) 
g(n) = o(-n). (2.203) 


Next consider the multinomial distribution that, for a single observation x, takes 


the form 
M M 
P(x|H) = II Hg” = exp 2 £p ln m) (2.204) 
k=1 k=1 


where x = (£1,..., £y)". Again, we can write this in the standard representation 
(2.194) so that 


p(x|n) = exp(n"x) (2.205) 
where nk = ln up, and we have defined n = (m,... nm)”. Again, comparing with 
(2.194) we have 

u(x) = x (2.206) 
h(x) = 1 (2.207) 
gn) = 1. (2.208) 


Note that the parameters ny are not independent because the parameters yẹ are sub- 
ject to the constraint 


ee (2.209) 


so that, given any M — 1 of the parameters ux, the value of the remaining parameter 
is fixed. In some circumstances, it will be convenient to remove this constraint by 
expressing the distribution in terms of only M — 1 parameters. This can be achieved 
by using the relationship (2.209) to eliminate um by expressing it in terms of the 
remaining {up} where k = 1,..., M — 1, thereby leaving M — 1 parameters. Note 
that these remaining parameters are still subject to the constraints 


O< py <1, et (2.210) 
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Making use of the constraint (2.209), the multinomial distribution in this representa- 
tion then becomes 


M 
exp [Samm] 
k=1 
M-1 M-1 M-1 
on £k ln Uk + (: — > n) In (: — > m) 
k=1 k=1 k=1 


M-1 M-1 
= exp `> ty, In ( La ) + In (: 5 m) 
k=1 q= Jya Hj k=1 


We now identify 


Hk 
l ——— = 2.212 
n (; = a =) Nk ( ) 


which we can solve for ug by first summing both sides over k and then rearranging 
and back-substituting to give 


_ _ exp(7x) 
ik= 
L ao exp(nz) 


This is called the softmax function, or the normalized exponential. In this represen- 
tation, the multinomial distribution therefore takes the form 


(2.213) 


M-1 =i 
p(x|n) = (: A > cot) exp(7' x). (2.214) 
k=l 


This is the standard form of the exponential family, with parameter vector 7 = 
(m, ..-,nm—1)" in which 


u(x) = x (2.215) 
h(x) = 1 (2.216) 


M-1 -1 
[: i 3 eatu) , (2.217) 
k=1 


Finally, let us consider the Gaussian distribution. For the univariate Gaussian, 
we have 


Q 
eN 
3 
Na 
I 


1 1 
plz|u, o’) = TE OP { 552 uy} (2.218) 


1 l 2, Æ 1 o 
-= TER] 552% +t sa} (2.219) 
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which, after some simple rearrangement, can be cast in the standard exponential 


Exercise 2.57 family form (2.194) with 
2 

n = (4/202) (2.220) 
u(x) = (2) (2.221) 
h(x) = (2m)? (2.222) 

2 
g(n) = (—2n2)'/? exp (#4). (2.223) 

Ane 


2.4.1 Maximum likelihood and sufficient statistics 


Let us now consider the problem of estimating the parameter vector 77 in the gen- 
eral exponential family distribution (2.194) using the technique of maximum likeli- 
hood. Taking the gradient of both sides of (2.195) with respect to 7, we have 


Va(n) I h(x) exp a 


+ g(n) / h(x) exp {7"u(x) } u(x) dx = 0. (2.224) 


Rearranging, and making use again of (2.195) then gives 
1 
g(n) 


where we have used (2.194). We therefore obtain the result 


Va(n) = g(n) f h(x) exp {n"u(x)} u(x) dx = E[u(x)] (2.225) 


—V ln g(n) = E[u(x)]. (2.226) 


Note that the covariance of u(x) can be expressed in terms of the second derivatives 
Exercise 2.58 of g(7), and similarly for higher order moments. Thus, provided we can normalize a 
distribution from the exponential family, we can always find its moments by simple 
differentiation. 
Now consider a set of independent identically distributed data denoted by X = 
{X1,.--,Xn}, for which the likelihood function is given by 


p(X|n) = [IL se) son "al PS u(Xn) . (2.227) 


Setting the gradient of In p(X|7) with respect to 7 to zero, we get the following 
condition to be satisfied by the maximum likelihood estimator Nm, 


-V Ing(nu) = = > u(x) (2.228) 
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which can in principle be solved to obtain Nnm. We see that the solution for the 
maximum likelihood estimator depends on the data only through `, u(x), which 
is therefore called the sufficient statistic of the distribution (2.194). We do not need 
to store the entire data set itself but only the value of the sufficient statistic. For 
the Bernoulli distribution, for example, the function u(x) is given just by x and 
so we need only keep the sum of the data points {£n}, whereas for the Gaussian 
u(x) = (x, x7)", and so we should keep both the sum of {z,,} and the sum of {z2 }. 

If we consider the limit N — oo, then the right-hand side of (2.228) becomes 
E/u(x)], and so by comparing with (2.226) we see that in this limit nmz, will equal 
the true value 77. 

In fact, this sufficiency property holds also for Bayesian inference, although 
we shall defer discussion of this until Chapter 8 when we have equipped ourselves 
with the tools of graphical models and can thereby gain a deeper insight into these 
important concepts. 


2.4.2 Conjugate priors 


We have already encountered the concept of a conjugate prior several times, for 
example in the context of the Bernoulli distribution (for which the conjugate prior 
is the beta distribution) or the Gaussian (where the conjugate prior for the mean is 
a Gaussian, and the conjugate prior for the precision is the Wishart distribution). In 
general, for a given probability distribution p(x|7), we can seek a prior p(n) that is 
conjugate to the likelihood function, so that the posterior distribution has the same 
functional form as the prior. For any member of the exponential family (2.194), there 
exists a conjugate prior that can be written in the form 


P(n|x,v) = F(x vgn)” exp {un* x} (2.229) 


where f(x, 1) is a normalization coefficient, and g(7) is the same function as ap- 
pears in (2.194). To see that this is indeed conjugate, let us multiply the prior (2.229) 
by the likelihood function (2.227) to obtain the posterior distribution, up to a nor- 
malization coefficient, in the form 


N 
p(X, x, v) x g(m)’*% exp far (>: u(Xn) + x) | (2.230) 
n=1 


This again takes the same functional form as the prior (2.229), confirming conjugacy. 
Furthermore, we see that the parameter v can be interpreted as a effective number of 
pseudo-observations in the prior, each of which has a value for the sufficient statistic 
u(x) given by x. 


2.4.3 Noninformative priors 


In some applications of probabilistic inference, we may have prior knowledge 
that can be conveniently expressed through the prior distribution. For example, if 
the prior assigns zero probability to some value of variable, then the posterior dis- 
tribution will necessarily also assign zero probability to that value, irrespective of 
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any subsequent observations of data. In many cases, however, we may have little 
idea of what form the distribution should take. We may then seek a form of prior 
distribution, called a noninformative prior, which is intended to have as little influ- 
ence on the posterior distribution as possible (Jeffries, 1946; Box and Tao, 1973; 
Bernardo and Smith, 1994). This is sometimes referred to as ‘letting the data speak 
for themselves’. 

If we have a distribution p(|) governed by a parameter A, we might be tempted 
to propose a prior distribution p(A) = const as a suitable prior. If A is a discrete 
variable with K states, this simply amounts to setting the prior probability of each 
state to 1/4. In the case of continuous parameters, however, there are two potential 
difficulties with this approach. The first is that, if the domain of À is unbounded, 
this prior distribution cannot be correctly normalized because the integral over A 
diverges. Such priors are called improper. In practice, improper priors can often 
be used provided the corresponding posterior distribution is proper, i.e., that it can 
be correctly normalized. For instance, if we put a uniform prior distribution over 
the mean of a Gaussian, then the posterior distribution for the mean, once we have 
observed at least one data point, will be proper. 

A second difficulty arises from the transformation behaviour of a probability 
density under a nonlinear change of variables, given by (1.27). If a function h(A) 
is constant, and we change variables to A = 7’, then h(n) = h(n?) will also be 
constant. However, if we choose the density p) (A) to be constant, then the density 
of 7 will be given, from (1.27), by 


Pn(n) = pata) Ea = py(n°)2n x n (2.231) 
and so the density over ņ will not be constant. This issue does not arise when we use 
maximum likelihood, because the likelihood function p(z|A) is a simple function of 
A and so we are free to use any convenient parameterization. If, however, we are to 
choose a prior distribution that is constant, we must take care to use an appropriate 
representation for the parameters. 

Here we consider two simple examples of noninformative priors (Berger, 1985). 
First of all, if a density takes the form 


p(x|u) = f(x — u) (2.232) 


then the parameter u is known as a location parameter. This family of densities 
exhibits translation invariance because if we shift x by a constant to give £ = £ +c, 
then 

pen) = f(@ - P) (2.233) 


where we have defined ji = u + c. Thus the density takes the same form in the 
new variable as in the original one, and so the density is independent of the choice 
of origin. We would like to choose a prior distribution that reflects this translation 
invariance property, and so we choose a prior that assigns equal probability mass to 
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an interval A < u < Bas to the shifted interval A — c < u < B — c. This implies 


B B-c B 
1 p(u) du = | plu) du = J p(u— cap (2.234) 


A A-c 


and because this must hold for all choices of A and B, we have 


plu — c) = ple) (2.235) 


which implies that p(j.) is constant. An example of a location parameter would be 
the mean u of a Gaussian distribution. As we have seen, the conjugate prior distri- 
bution for y in this case is a Gaussian p(u|uo, 06) = N (u| uo, c), and we obtain a 
noninformative prior by taking the limit o — oo. Indeed, from (2.141) and (2.142) 
we see that this gives a posterior distribution over u in which the contributions from 
the prior vanish. 

As a second example, consider a density of the form 


P(z|o) = i (=) (2.236) 


where ø > 0. Note that this will be a normalized density provided f(x) is correctly 
normalized. The parameter ø is known as a scale parameter, and the density exhibits 
scale invariance because if we scale x by a constant to give 7 = cz, then 


~ ke 
p(a|o) = at (=) (2.237) 


where we have defined ¢ = co. This transformation corresponds to a change of 
scale, for example from meters to kilometers if x is a length, and we would like 
to choose a prior distribution that reflects this scale invariance. If we consider an 
interval A < ø < B, anda scaled interval A/c < o < B/c, then the prior should 
assign equal probability mass to these two intervals. Thus we have 


B B/c B AN] 
| p(a) do = i plo)do = | p (2°) — do (2.238) 
A A/c A c c 


and because this must hold for choices of A and B, we have 
plo) =p (že) = (2.239) 


and hence p(o) x 1/o. Note that again this is an improper prior because the integral 
of the distribution over 0 < ø < œ is divergent. It is sometimes also convenient 
to think of the prior distribution for a scale parameter in terms of the density of the 
log of the parameter. Using the transformation rule (1.27) for densities we see that 
p(In a) = const. Thus, for this prior there is the same probability mass in the range 
1 < ø < 10 as in the range 10 < ø < 100 and in 100 < ø < 1000. 
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2.5. 


An example of a scale parameter would be the standard deviation o of a Gaussian 
distribution, after we have taken account of the location parameter u, because 


N(z|u, 0°) x o~* exp {—(Z/a)?} (2.240) 


where x = x — u. As discussed earlier, it is often more convenient to work in terms 
of the precision A = 1/o? rather than ø itself. Using the transformation rule for 
densities, we see that a distribution p(o) œx 1/o corresponds to a distribution over À 
of the form p(A) œx 1/A. We have seen that the conjugate prior for A was the gamma 
distribution Gam(A|ao, bo) given by (2.146). The noninformative prior is obtained 
as the special case ao = bp = 0. Again, if we examine the results (2.150) and (2.151) 
for the posterior distribution of A, we see that for a9 = bo = 0, the posterior depends 
only on terms arising from the data and not from the prior. 


Nonparametric Methods 


Throughout this chapter, we have focussed on the use of probability distributions 
having specific functional forms governed by a small number of parameters whose 
values are to be determined from a data set. This is called the parametric approach 
to density modelling. An important limitation of this approach is that the chosen 
density might be a poor model of the distribution that generates the data, which can 
result in poor predictive performance. For instance, if the process that generates the 
data is multimodal, then this aspect of the distribution can never be captured by a 
Gaussian, which is necessarily unimodal. 

In this final section, we consider some nonparametric approaches to density es- 
timation that make few assumptions about the form of the distribution. Here we shall 
focus mainly on simple frequentist methods. The reader should be aware, however, 
that nonparametric Bayesian methods are attracting increasing interest (Walker et al., 
1999; Neal, 2000; Muller and Quintana, 2004; Teh et al., 2006). 

Let us start with a discussion of histogram methods for density estimation, which 
we have already encountered in the context of marginal and conditional distributions 
in Figure 1.11 and in the context of the central limit theorem in Figure 2.6. Here we 
explore the properties of histogram density models in more detail, focussing on the 
case of a single continuous variable x. Standard histograms simply partition x into 
distinct bins of width A; and then count the number n; of observations of x falling 
in bin 2. In order to turn this count into a normalized probability density, we simply 
divide by the total number N of observations and by the width A; of the bins to 
obtain probability values for each bin given by 


Ni 


~ NA; 


Pi (2.241) 
for which it is easily seen that f p(x)dx = 1. This gives a model for the density 


p(x) that is constant over the width of each bin, and often the bins are chosen to have 
the same width A; = A. 


Figure 2.24 
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An illustration of the histogram approach 5 
to density estimation, in which a data set A = 0.04 
of 50 data points is generated from the 
distribution shown by the green curve. 
Histogram density estimates, based on `o 0.5 1l 
(2.241), with a common bin width A are ; 


5 f 
shown for various values of A. A = 0.08 
0 


5° 0.5 1 
A = 0.25 
0 
0 0.5 1 


In Figure 2.24, we show an example of histogram density estimation. Here 
the data is drawn from the distribution, corresponding to the green curve, which is 
formed from a mixture of two Gaussians. Also shown are three examples of his- 
togram density estimates corresponding to three different choices for the bin width 
A. We see that when A is very small (top figure), the resulting density model is very 
spiky, with a lot of structure that is not present in the underlying distribution that 
generated the data set. Conversely, if A is too large (bottom figure) then the result is 
a model that is too smooth and that consequently fails to capture the bimodal prop- 
erty of the green curve. The best results are obtained for some intermediate value 
of A (middle figure). In principle, a histogram density model is also dependent on 
the choice of edge location for the bins, though this is typically much less significant 
than the value of A. 

Note that the histogram method has the property (unlike the methods to be dis- 
cussed shortly) that, once the histogram has been computed, the data set itself can 
be discarded, which can be advantageous if the data set is large. Also, the histogram 
approach is easily applied if the data points are arriving sequentially. 

In practice, the histogram technique can be useful for obtaining a quick visual- 
ization of data in one or two dimensions but is unsuited to most density estimation 
applications. One obvious problem is that the estimated density has discontinuities 
that are due to the bin edges rather than any property of the underlying distribution 
that generated the data. Another major limitation of the histogram approach is its 
scaling with dimensionality. If we divide each variable in a D-dimensional space 
into M bins, then the total number of bins will be MP. This exponential scaling 
with D is an example of the curse of dimensionality. In a space of high dimensional- 
ity, the quantity of data needed to provide meaningful estimates of local probability 
density would be prohibitive. 

The histogram approach to density estimation does, however, teach us two im- 
portant lessons. First, to estimate the probability density at a particular location, 
we should consider the data points that lie within some local neighbourhood of that 
point. Note that the concept of locality requires that we assume some form of dis- 
tance measure, and here we have been assuming Euclidean distance. For histograms, 
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this neighbourhood property was defined by the bins, and there is a natural ‘smooth- 
ing’ parameter describing the spatial extent of the local region, in this case the bin 
width. Second, the value of the smoothing parameter should be neither too large nor 
too small in order to obtain good results. This is reminiscent of the choice of model 
complexity in polynomial curve fitting discussed in Chapter 1 where the degree M 
of the polynomial, or alternatively the value a of the regularization parameter, was 
optimal for some intermediate value, neither too large nor too small. Armed with 
these insights, we turn now to a discussion of two widely used nonparametric tech- 
niques for density estimation, kernel estimators and nearest neighbours, which have 
better scaling with dimensionality than the simple histogram model. 


2.5.1 Kernel density estimators 


Let us suppose that observations are being drawn from some unknown probabil- 
ity density p(x) in some D-dimensional space, which we shall take to be Euclidean, 
and we wish to estimate the value of p(x). From our earlier discussion of locality, 
let us consider some small region R containing x. The probability mass associated 
with this region is given by 


P= | p(x) dx. (2.242) 
R 
Now suppose that we have collected a data set comprising N observations drawn 
from p(x). Because each data point has a probability P of falling within R, the total 
number K of points that lie inside R will be distributed according to the binomial 
distribution wi 
: : K 1-K 

Bin(K|N, P) = KN- Kyi (1 — P) : (2.243) 
Using (2.11), we see that the mean fraction of points falling inside the region is 
E[K/N] = P, and similarly using (2.12) we see that the variance around this mean 
is var| K/N] = P(1 — P)/N. For large N, this distribution will be sharply peaked 
around the mean and so 


K ~ NP. (2.244) 


If, however, we also assume that the region R is sufficiently small that the probability 
density p(x) is roughly constant over the region, then we have 


P ~ p(x)V (2.245) 


where V is the volume of R. Combining (2.244) and (2.245), we obtain our density 


estimate in the form 
= K- 2.246 


Note that the validity of (2.246) depends on two contradictory assumptions, namely 
that the region R be sufficiently small that the density is approximately constant over 
the region and yet sufficiently large (in relation to the value of that density) that the 
number K of points falling inside the region is sufficient for the binomial distribution 
to be sharply peaked. 
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We can exploit the result (2.246) in two different ways. Either we can fix K and 
determine the value of V from the data, which gives rise to the K-nearest-neighbour 
technique discussed shortly, or we can fix V and determine K from the data, giv- 
ing rise to the kernel approach. It can be shown that both the K’-nearest-neighbour 
density estimator and the kernel density estimator converge to the true probability 
density in the limit N — oo provided V shrinks suitably with N, and K grows with 
N (Duda and Hart, 1973). 

We begin by discussing the kernel method in detail, and to start with we take 
the region R to be a small hypercube centred on the point x at which we wish to 
determine the probability density. In order to count the number K of points falling 
within this region, it is convenient to define the following function 


Ofi, el <7, i=1,...,D, 
klu) = { 0, otherwise (ea) 


which represents a unit cube centred on the origin. The function k(u) is an example 
of a kernel function, and in this context is also called a Parzen window. From (2.247), 
the quantity k((x — x,,)/h) will be one if the data point x, lies inside a cube of side 
h centred on x, and zero otherwise. The total number of data points lying inside this 


cube will therefore be N 
X— Xp, 
K= 2 k ( - ) l (2.248) 


Substituting this expression into (2.246) then gives the following result for the esti- 
mated density at x 


N 
p(x) = = 3 D k (= —*) (2.249) 


where we have used V = h?” for the volume of a hypercube of side h in D di- 
mensions. Using the symmetry of the function k(u), we can now re-interpret this 
equation, not as a single cube centred on x but as the sum over N cubes centred on 
the N data points xn. 

As it stands, the kernel density estimator (2.249) will suffer from one of the same 
problems that the histogram method suffered from, namely the presence of artificial 
discontinuities, in this case at the boundaries of the cubes. We can obtain a smoother 
density model if we choose a smoother kernel function, and a common choice is the 
Gaussian, which gives rise to the following kernel density model 


153 |x = xal]? a 
p(x) N >, (2rh2)!/2 exp 2h2 ( s ) 


where h represents the standard deviation of the Gaussian components. Thus our 
density model is obtained by placing a Gaussian over each data point and then adding 
up the contributions over the whole data set, and then dividing by N so that the den- 
sity is correctly normalized. In Figure 2.25, we apply the model (2.250) to the data 
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Figure 2.25 


Illustration of the kernel density model 5 


(2.250) applied to the same data set used h = 0.005 | 
to demonstrate the histogram approach in 
Figure 2.24. We see that h acts as a 0 


smoothing parameter and that if it is set 0 0.5 1 
too small (top panel), the result is a very 5 : 

noisy density model, whereas if it is set h = 0.07 

too large (bottom panel), then the bimodal a eee ae 
nature of the underlying distribution from 0 


which the data is generated (shown by the “9 0.5 1 
green curve) is washed out. The best den- 5 
sity model is obtained for some intermedi- 


ate value of h (middle panel). Z N 
0 


set used earlier to demonstrate the histogram technique. We see that, as expected, 
the parameter h plays the role of a smoothing parameter, and there is a trade-off 
between sensitivity to noise at small h and over-smoothing at large h. Again, the 
optimization of h is a problem in model complexity, analogous to the choice of bin 
width in histogram density estimation, or the degree of the polynomial used in curve 
fitting. 

We can choose any other kernel function k(u) in (2.249) subject to the condi- 
tions 


k(u) > 0, (2.251) 
l k(u)du = 1 (2.252) 


which ensure that the resulting probability distribution is nonnegative everywhere 
and integrates to one. The class of density model given by (2.249) is called a kernel 
density estimator, or Parzen estimator. It has a great merit that there is no compu- 
tation involved in the ‘training’ phase because this simply requires storage of the 
training set. However, this is also one of its great weaknesses because the computa- 
tional cost of evaluating the density grows linearly with the size of the data set. 


2.5.2 Nearest-neighbour methods 


One of the difficulties with the kernel approach to density estimation is that the 
parameter h governing the kernel width is fixed for all kernels. In regions of high 
data density, a large value of h may lead to over-smoothing and a washing out of 
structure that might otherwise be extracted from the data. However, reducing h may 
lead to noisy estimates elsewhere in data space where the density is smaller. Thus 
the optimal choice for h may be dependent on location within the data space. This 
issue is addressed by nearest-neighbour methods for density estimation. 

We therefore return to our general result (2.246) for local density estimation, 
and instead of fixing V and determining the value of K from the data, we consider 
a fixed value of K and use the data to find an appropriate value for V. To do this, 
we consider a small sphere centred on the point x at which we wish to estimate the 


Figure 2.26 Illustration of K-nearest-neighbour den- 5 


Exercise 2.61 


2.5. Nonparametric Methods 125 


sity estimation using the same data set K=1 
as in Figures 2.25 and 2.24. We see 

that the parameter K governs the degree 

of smoothing, so that a small value of “5 0.5 1 
K leads to a very noisy density model 5 . 
(top panel), whereas a large value (bot- K=5 
tom panel) smoothes out the bimodal na- 
ture of the true distribution (shown by the 
green curve) from which the data set was “9 0.5 1 
generated. 5 


0 0.5 1 


density p(x), and we allow the radius of the sphere to grow until it contains precisely 
K data points. The estimate of the density p(x) is then given by (2.246) with V set to 
the volume of the resulting sphere. This technique is known as K nearest neighbours 
and is illustrated in Figure 2.26, for various choices of the parameter K, using the 
same data set as used in Figure 2.24 and Figure 2.25. We see that the value of K 
now governs the degree of smoothing and that again there is an optimum choice for 
K that is neither too large nor too small. Note that the model produced by K nearest 
neighbours is not a true density model because the integral over all space diverges. 

We close this chapter by showing how the /’-nearest-neighbour technique for 
density estimation can be extended to the problem of classification. To do this, we 
apply the K-nearest-neighbour density estimation technique to each class separately 
and then make use of Bayes’ theorem. Let us suppose that we have a data set com- 
prising V;, points in class C with N points in total, so that X`, Ng = N. If we 
wish to classify a new point x, we draw a sphere centred on x containing precisely 
K points irrespective of their class. Suppose this sphere has volume V and contains 
Kķņ points from class Cx. Then (2.246) provides an estimate of the density associated 
with each class 


Ky 
= 22 
p(x|Ck) N,V (2.253) 
Similarly, the unconditional density is given by 
K 
= — 2.254 
p(x) = Jy (2.254) 
while the class priors are given by 
N; 
pC) = >> (2.255) 


We can now combine (2.253), (2.254), and (2.255) using Bayes’ theorem to obtain 
the posterior probability of class membership 
P(X|Ce)P(Ce) _ Kx 


p(Cr|x) = rey =a (2.256) 
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Figure 2.27 (a) In the K-nearest- 
neighbour classifier, a new point, 
shown by the black diamond, is clas- 
sified according to the majority class 
membership of the K closest train- 
ing data points, in this case K = 
3. (b) In the nearest-neighbour 
(K = 1) approach to classification, 
the resulting decision boundary is 
composed of hyperplanes that form 
perpendicular bisectors of pairs of 
points from different classes. 
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Ti 


(b) 
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If we wish to minimize the probability of misclassification, this is done by assigning 
the test point x to the class having the largest posterior probability, corresponding to 
the largest value of K/K. Thus to classify a new point, we identify the K nearest 
points from the training data set and then assign the new point to the class having the 
largest number of representatives amongst this set. Ties can be broken at random. 
The particular case of K = 1 is called the nearest-neighbour rule, because a test 
point is simply assigned to the same class as the nearest point from the training set. 


These concepts are illustrated in Figure 2.27. 


In Figure 2.28, we show the results of applying the K-nearest-neighbour algo- 
rithm to the oil flow data, introduced in Chapter 1, for various values of K. As 
expected, we see that K controls the degree of smoothing, so that small produces 
many small regions of each class, whereas large K leads to fewer larger regions. 
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Figure 2.28 Plot of 200 data points from the oil data set showing values of xs plotted against «7, where the 
red, green, and blue points correspond to the ‘laminar’, ‘annular’, and ‘homogeneous’ classes, respectively. Also 
shown are the classifications of the input space given by the K-nearest-neighbour algorithm for various values 


of K. 
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An interesting property of the nearest-neighbour (K = 1) classifier is that, in the 
limit NV — oo, the error rate is never more than twice the minimum achievable error 
rate of an optimal classifier, i.e., one that uses the true class distributions (Cover and 
Hart, 1967) . 

As discussed so far, both the /’-nearest-neighbour method, and the kernel den- 
sity estimator, require the entire training data set to be stored, leading to expensive 
computation if the data set is large. This effect can be offset, at the expense of some 
additional one-off computation, by constructing tree-based search structures to allow 
(approximate) near neighbours to be found efficiently without doing an exhaustive 
search of the data set. Nevertheless, these nonparametric methods are still severely 
limited. On the other hand, we have seen that simple parametric models are very 
restricted in terms of the forms of distribution that they can represent. We therefore 
need to find density models that are very flexible and yet for which the complexity 
of the models can be controlled independently of the size of the training set, and we 
shall see in subsequent chapters how to achieve this. 


œ FWY Verify that the Bernoulli distribution (2.2) satisfies the following prop- 
erties 


1 
D OE (2.257) 
x=0 
Elz] = p (2.258) 
var[z] = p(l— yp). (2.259) 


Show that the entropy H[z] of a Bernoulli distributed random binary variable x is 
given by 
Hfz] = -un p — (1 — u) In(1 — u). (2.260) 


(xx) The form of the Bernoulli distribution given by (2.2) is not symmetric be- 
tween the two values of x. In some situations, it will be more convenient to use an 
equivalent formulation for which z € {—1, 1}, in which case the distribution can be 


meee (1—2)/2 (142)/2 
1— ~~ 1+ 7 
p(z|u) = (4 r) oa r) (2.261) 


where u € [—1, 1]. Show that the distribution (2.261) is normalized, and evaluate its 
mean, variance, and entropy. 


œ» [QTY In this exercise, we prove that the binomial distribution (2.9) is nor- 
malized. First use the definition (2.10) of the number of combinations of m identical 
objects chosen from a total of N to show that 


(+ (= GD) oe 
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Use this result to prove by induction the following result 


N 
G+) => (>) a” (2.263) 


m=0 


which is known as the binomial theorem, and which is valid for all real values of x. 
Finally, show that the binomial distribution is normalized, so that 


N 


> (y) wip Si (2.264) 
m 


m=0 


which can be done by first pulling out a factor (1 — p)™ out of the summation and 
then making use of the binomial theorem. 


2.4 (xx) Show that the mean of the binomial distribution is given by (2.11). To do this, 
differentiate both sides of the normalization condition (2.264) with respect to u and 
then rearrange to obtain an expression for the mean of n. Similarly, by differentiating 
(2.264) twice with respect to u and making use of the result (2.11) for the mean of 
the binomial distribution prove the result (2.12) for the variance of the binomial. 


2.5 œ» [QQ In this exercise, we prove that the beta distribution, given by (2.13), is 
correctly normalized, so that (2.14) holds. This is equivalent to showing that 


Í vita du = Tee (2.265) 


From the definition (1.141) of the gamma function, we have 


T(a)P(b) = i exp(—2x)x2°* a f exp(—y)y? 1 dy. (2.266) 
0 0 


Use this expression to prove (2.265) as follows. First bring the integral over y inside 
the integrand of the integral over x, next make the change of variable t = y + x 
where z7 is fixed, then interchange the order of the x and t integrations, and finally 
make the change of variable x = tu where t is fixed. 


2.6 (x) Make use of the result (2.265) to show that the mean, variance, and mode of the 
beta distribution (2.13) are given respectively by 


Eju] = a] (2.267) 
ab 

ven] (a+b)2(a+b+1) aana 

moii s -e (2.269) 


a+b-2 
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(xx) Consider a binomial random variable x given by (2.9), with prior distribution 
for js given by the beta distribution (2.13), and suppose we have observed m occur- 
rences of x = 1 and Į occurrences of x = 0. Show that the posterior mean value of x 
lies between the prior mean and the maximum likelihood estimate for u. To do this, 
show that the posterior mean can be written as times the prior mean plus (1 — A) 
times the maximum likelihood estimate, where 0 < A < 1. This illustrates the con- 
cept of the posterior distribution being a compromise between the prior distribution 
and the maximum likelihood solution. 


(x) Consider two variables x and y with joint distribution p(x, y). Prove the follow- 
ing two results 


E[z] = E, [E,[z\y]] (2.270) 
var[z] = E, [vars[z|y]] + vary [E+ [xy]. (2.271) 


Here E,,[x|y] denotes the expectation of x under the conditional distribution p(z|y), 
with a similar notation for the conditional variance. 


(xxx) www | . In this exercise, we prove the normalization of the Dirichlet dis- 
tribution (2.38) using induction. We have already shown in Exercise 2.5 that the 
beta distribution, which is a special case of the Dirichlet for M = 2, is normalized. 
We now assume that the Dirichlet distribution is normalized for M — 1 variables 
and prove that it is normalized for M variables. To do this, consider the Dirichlet 
distribution over M variables, and take account of the constraint ye Le = 1 by 
eliminating um, so that the Dirichlet is written 


M-1 M-1 om 1 
pa(tay-»-5fa—1) = Cm || gt [: -> m) (2.272) 
k=1 j=1 


and our goal is to find an expression for Cm. To do this, integrate over y m-—1, taking 
care over the limits of integration, and then make a change of variable so that this 
integral has limits 0 and 1. By assuming the correct result for C')7_; and making use 
of (2.265), derive the expression for Cm. 


(xx) Using the property T(x + 1) = aI (x) of the gamma function, derive the 
following results for the mean, variance, and covariance of the Dirichlet distribution 
given by (2.38) 


E[u,j] = a (2.273) 
Qi Qo — Aj 
var [ju5] n (ao 7 i (2.274) 
aya ` 
cov[ujm] = aeo PAI (2.275) 
0 


where a is defined by (2.39). 
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2.11 


2.12 


2.13 


2.14 


2.15 


(x) FY By expressing the expectation of In jz; under the Dirichlet distribution 
(2.38) as a derivative with respect to œj, show that 


Eln uj] = Y(a;) — (ao) (2.276) 


where ao is given by (2.39) and 


wa) = 4 


gero (2.277) 


is the digamma function. 


(x) The uniform distribution for a continuous variable x is defined by 


1 


U(ala, b) = a 


axga<b. (2.278) 


Verify that this distribution is normalized, and find expressions for its mean and 
variance. 


(xx) Evaluate the Kullback-Leibler divergence (1.113) between two Gaussians 
p(x) = N(x|u, X) and q(x) = N (x|m, L). 


œ» EANA This exercise demonstrates that the multivariate distribution with max- 
imum entropy, for a given covariance, is a Gaussian. The entropy of a distribution 
p(x) is given by 


H[x] = — frw ln p(x) dx. (2.279) 


We wish to maximize H[x] over all distributions p(x) subject to the constraints that 
p(x) be normalized and that it have a specific mean and covariance, so that 


f p(x) dx = 1 (2.280) 
f v00xax =p (2.281) 
froe — p)(x— p)” dx = 5. (2.282) 


By performing a variational maximization of (2.279) and using Lagrange multipliers 
to enforce the constraints (2.280), (2.281), and (2.282), show that the maximum 
likelihood distribution is given by the Gaussian (2.43). 


(xx) Show that the entropy of the multivariate Gaussian N (x| u, ©) is given by 


D 


z (1+ In(2z)) (2.283) 


Ax] = in [S| 
2 


where D is the dimensionality of x. 


2.16 


2.17 


2.18 


2.19 


2.20 


2.21 
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(xxx) www | Consider two random variables 7, and x2 having Gaussian distri- 
butions with means 41, 42 and precisions T1, T2 respectively. Derive an expression 
for the differential entropy of the variable x = xı + x2. To do this, first find the 
distribution of x by using the relation 


p(x) = in p(a|x2)p(x2) dre (2.284) 


CO 


and completing the square in the exponent. Then observe that this represents the 
convolution of two Gaussian distributions, which itself will be Gaussian, and finally 
make use of the result (1.110) for the entropy of the univariate Gaussian. 


œ FQ Consider the multivariate Gaussian distribution given by (2.43). By 
writing the precision matrix (inverse covariance matrix) ~' as the sum of a sym- 
metric and an anti-symmetric matrix, show that the anti-symmetric term does not 
appear in the exponent of the Gaussian, and hence that the precision matrix may be 
taken to be symmetric without loss of generality. Because the inverse of a symmetric 
matrix is also symmetric (see Exercise 2.22), it follows that the covariance matrix 
may also be chosen to be symmetric without loss of generality. 


(xxx) Consider a real, symmetric matrix © whose eigenvalue equation is given 
by (2.45). By taking the complex conjugate of this equation and subtracting the 
original equation, and then forming the inner product with eigenvector u;, show that 
the eigenvalues A; are real. Similarly, use the symmetry property of X to show that 
two eigenvectors u; and u; will be orthogonal provided A; # A;. Finally, show that 
without loss of generality, the set of eigenvectors can be chosen to be orthonormal, 
so that they satisfy (2.46), even if some of the eigenvalues are zero. 


(xx) Show that a real, symmetric matrix & having the eigenvector equation (2.45) 
can be expressed as an expansion in the eigenvectors, with coefficients given by the 
eigenvalues, of the form (2.48). Similarly, show that the inverse matrix X~" has a 
representation of the form (2.49). 


(« x) | www | A positive definite matrix & can be defined as one for which the 
quadratic form 
ge 2 (2.285) 


is positive for any real value of the vector a. Show that a necessary and sufficient 
condition for & to be positive definite is that all of the eigenvalues A; of X, defined 
by (2.45), are positive. 


(x) Show that a real, symmetric matrix of size D x D has D(D + 1)/2 independent 
parameters. 


œ FEY Show that the inverse of a symmetric matrix is itself symmetric. 


(xx) By diagonalizing the coordinate system using the eigenvector expansion (2.45), 
show that the volume contained within the hyperellipsoid corresponding to a constant 
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2.24 


2.25 


2.26 


2.27 


2.28 


Mahalanobis distance A is given by 
Vora? (2.286) 


where Vp is the volume of the unit sphere in D dimensions, and the Mahalanobis 
distance is defined by (2.44). 


(x x) E Prove the identity (2.76) by multiplying both sides by the matrix 


A B 
(2 P) (2.287) 


and making use of the definition (2.77). 


(xx) In Sections 2.3.1 and 2.3.2, we considered the conditional and marginal distri- 
butions for a multivariate Gaussian. More generally, we can consider a partitioning 
of the components of x into three groups Xa, Xp, and x,, with a corresponding par- 
titioning of the mean vector yz and of the covariance matrix X in the form 


Ha Yaa Xab Zac 
u= Ha |, x= Mba Xb Dpc à (2.288) 
He Yea Deb Yee 


By making use of the results of Section 2.3, find an expression for the conditional 
distribution p(x,|x») in which x, has been marginalized out. 


(xx) A very useful result from linear algebra is the Woodbury matrix inversion 
formula given by 


(A + BCD)! = A~! — A`'B(C~! + DAB) 7! DAT. (2.289) 
By multiplying both sides by (A + BCD) prove the correctness of this result. 


(x) Let x and z be two independent random vectors, so that p(x,z) = p(x)p(z). 
Show that the mean of their sum y = x + z is given by the sum of the means of each 
of the variable separately. Similarly, show that the covariance matrix of y is given by 
the sum of the covariance matrices of x and z. Confirm that this result agrees with 
that of Exercise 1.10. 


œx» [QTY Consider a joint distribution over the variable 


= x 
z= () (2.290) 


whose mean and covariance are given by (2.108) and (2.105) respectively. By mak- 
ing use of the results (2.92) and (2.93) show that the marginal distribution p(x) is 
given (2.99). Similarly, by making use of the results (2.81) and (2.82) show that the 
conditional distribution p(y|x) is given by (2.100). 


2.29 


2.30 


2.31 


2.32 


2.33 


2.34 


2.35 


2.36 
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(xx) Using the partitioned matrix inversion formula (2.76), show that the inverse of 
the precision matrix (2.104) is given by the covariance matrix (2.105). 


(x) By starting from (2.107) and making use of the result (2.105), verify the result 
(2.108). 


(xx) Consider two multidimensional random vectors x and z having Gaussian 
distributions p(x) = N(x|p,, Ux) and p(z) = N(z|u,, Xz) respectively, together 
with their sum y = x+z. Use the results (2.109) and (2.110) to find an expression for 
the marginal distribution p(y) by considering the linear-Gaussian model comprising 
the product of the marginal distribution p(x) and the conditional distribution p(y|x). 


œx» [QQ This exercise and the next provide practice at manipulating the 
quadratic forms that arise in linear-Gaussian models, as well as giving an indepen- 
dent check of results derived in the main text. Consider a joint distribution p(x, y) 
defined by the marginal and conditional distributions given by (2.99) and (2.100). 
By examining the quadratic form in the exponent of the joint distribution, and using 
the technique of “completing the square’ discussed in Section 2.3, find expressions 
for the mean and covariance of the marginal distribution p(y) in which the variable 
x has been integrated out. To do this, make use of the Woodbury matrix inversion 
formula (2.289). Verify that these results agree with (2.109) and (2.110) obtained 
using the results of Chapter 2. 


(xxx) Consider the same joint distribution as in Exercise 2.32, but now use the 
technique of completing the square to find expressions for the mean and covariance 
of the conditional distribution p(x|y). Again, verify that these agree with the corre- 
sponding expressions (2.111) and (2.112). 


œ» FQ To find the maximum likelihood solution for the covariance matrix 
of a multivariate Gaussian, we need to maximize the log likelihood function (2.118) 
with respect to ©, noting that the covariance matrix must be symmetric and positive 
definite. Here we proceed by ignoring these constraints and doing a straightforward 
maximization. Using the results (C.21), (C.26), and (C.28) from Appendix C, show 
that the covariance matrix X that maximizes the log likelihood function (2.118) is 
given by the sample covariance (2.122). We note that the final result is necessarily 
symmetric and positive definite (provided the sample covariance is nonsingular). 


(xx) Use the result (2.59) to prove (2.62). Now, using the results (2.59), and (2.62), 
show that 


E[XnXm] = HU” + Inm™ (2.291) 


where xn denotes a data point sampled from a Gaussian distribution with mean p 
and covariance X, and Inm denotes the (n, m) element of the identity matrix. Hence 
prove the result (2.124). 


(xx) FQ Using an analogous procedure to that used to obtain (2.126), derive 
an expression for the sequential estimation of the variance of a univariate Gaussian 
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2.37 


2.38 


2.39 


2.40 


2.41 


2.42 
2.43 


distribution, by starting with the maximum likelihood expression 


N 
1 
on = a Ga — u)’. (2.292) 
n=1 


Verify that substituting the expression for a Gaussian distribution into the Robbins- 
Monro sequential estimation formula (2.135) gives a result of the same form, and 
hence obtain an expression for the corresponding coefficients ay. 


(xx) Using an analogous procedure to that used to obtain (2.126), derive an ex- 
pression for the sequential estimation of the covariance of a multivariate Gaussian 
distribution, by starting with the maximum likelihood expression (2.122). Verify that 
substituting the expression for a Gaussian distribution into the Robbins-Monro se- 
quential estimation formula (2.135) gives a result of the same form, and hence obtain 
an expression for the corresponding coefficients ay. 


(x) Use the technique of completing the square for the quadratic form in the expo- 
nent to derive the results (2.141) and (2.142). 


(xx) Starting from the results (2.141) and (2.142) for the posterior distribution 
of the mean of a Gaussian random variable, dissect out the contributions from the 
first N — 1 data points and hence obtain expressions for the sequential update of 
un and a3. Now derive the same results starting from the posterior distribution 
P(ult1,---,@N-1) = N(plun—1,0%_1) and multiplying by the likelihood func- 
tion p(zy|u) = N (xy|u, 0°) and then completing the square and normalizing to 
obtain the posterior distribution after N observations. 


(« x) | www | Consider a D-dimensional Gaussian random variable x with distribu- 
tion N’(x|, ©) in which the covariance © is known and for which we wish to infer 
the mean u from a set of observations X = {x1,...,x}. Given a prior distribution 
p(w) = N (u| Ho, Uo), find the corresponding posterior distribution p(u|X). 


(x) Use the definition of the gamma function (1.141) to show that the gamma dis- 
tribution (2.146) is normalized. 


(xx) Evaluate the mean, variance, and mode of the gamma distribution (2.146). 


(x) The following distribution 


plz|o’, q) = 1 exp ial (2.293) 
2(20?)1/4T'(1/q) 20? 
is a generalization of the univariate Gaussian distribution. Show that this distribution 
is normalized so that 
f plz|o?,q)dz = 1 (2.294) 
=00 
and that it reduces to the Gaussian when q = 2. Consider a regression model in 
which the target variable is given by t = y(x,w) + € and € is a random noise 
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2.45 


2.46 
2.47 


2.48 


2.49 


2.50 


2.51 


Exercises 135 


variable drawn from the distribution (2.293). Show that the log likelihood function 
over w and o°, for an observed data set of input vectors X = {x1,...,Xw} and 
corresponding target variables t = (ti,..., tn)", is given by 


N 
1 N 
In p(t|X, w, 0?) = 3 S| lyn, w) — tn]? — z ne”) + const (2.295) 
n=1 


where ‘const’ denotes terms independent of both w and o°. Note that, as a function 
of w, this is the Lq error function considered in Section 1.5.5. 


(xx) Consider a univariate Gaussian distribution \V(x|,7~') having conjugate 
Gaussian-gamma prior given by (2.154), and a data set X = {x,...,2@y} of iid. 
observations. Show that the posterior distribution is also a Gaussian-gamma distri- 
bution of the same functional form as the prior, and write down expressions for the 
parameters of this posterior distribution. 


(x) Verify that the Wishart distribution defined by (2.155) is indeed a conjugate 
prior for the precision matrix of a multivariate Gaussian. 


(x) FEY Verify that evaluating the integral in (2.158) leads to the result (2.159). 


(x) FY Show that in the limit v — oo, the t-distribution (2.159) becomes a 
Gaussian. Hint: ignore the normalization coefficient, and simply look at the depen- 
dence on z. 


(x) By following analogous steps to those used to derive the univariate Student’s 
t-distribution (2.159), verify the result (2.162) for the multivariate form of the Stu- 
dent’s t-distribution, by marginalizing over the variable 7 in (2.161). Using the 
definition (2.161), show by exchanging integration variables that the multivariate 
t-distribution is correctly normalized. 


(xx) By using the definition (2.161) of the multivariate Student’s t-distribution as a 
convolution of a Gaussian with a gamma distribution, verify the properties (2.164), 
(2.165), and (2.166) for the multivariate t-distribution defined by (2.162). 


(x) Show that in the limit v — oo, the multivariate Student’s t-distribution (2.162) 
reduces to a Gaussian with mean yz and precision A. 


œ [QTY The various trigonometric identities used in the discussion of periodic 
variables in this chapter can be proven easily from the relation 


exp(tA) = cos A + isin A (2.296) 
in which 7 is the square root of minus one. By considering the identity 
exp(iA) exp(—iA) = 1 (2.297) 
prove the result (2.177). Similarly, using the identity 
cos(A — B) = Rexp{i(A — B)} (2.298) 
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2.52 


2.53 


2.54 


2.55 


2.56 


2.57 


2.58 


2.59 


2.60 


2.61 


where R denotes the real part, prove (2.178). Finally, by using sin(A — B) = 
Sexp{i(A — B)}, where S denotes the imaginary part, prove the result (2.183). 


(xx) For large m, the von Mises distribution (2.179) becomes sharply peaked 
around the mode fo. By defining € = m'/?(@ — 0o) and making the Taylor ex- 
pansion of the cosine function given by 


2 


cosa = 1— F + O(a*) (2.299) 


show that as m — ov, the von Mises distribution tends to a Gaussian. 


(x) Using the trigonometric identity (2.183), show that solution of (2.182) for ĝo is 
given by (2.184). 


(x) By computing first and second derivatives of the von Mises distribution (2.179), 
and using I[o(m) > 0 for m > 0, show that the maximum of the distribution occurs 
when @ = 0o and that the minimum occurs when 0 = 69 + 7 (mod 27). 


(x) By making use of the result (2.168), together with (2.184) and the trigonometric 
identity (2.178), show that the maximum likelihood solution mmz for the concentra- 
tion of the von Mises distribution satisfies A(mmL) = T where F is the radius of the 
mean of the observations viewed as unit vectors in the two-dimensional Euclidean 
plane, as illustrated in Figure 2.17. 


œ» FY Express the beta distribution (2.13), the gamma distribution (2.146), 
and the von Mises distribution (2.179) as members of the exponential family (2.194) 
and thereby identify their natural parameters. 


(x) Verify that the multivariate Gaussian distribution can be cast in exponential 
family form (2.194) and derive expressions for 7, u(x), h(x) and g(7) analogous to 
(2.220)-(2.223). 


(x) The result (2.226) showed that the negative gradient of In g(7) for the exponen- 
tial family is given by the expectation of u(x). By taking the second derivatives of 
(2.195), show that 


-VV Ing(n) = Efu(x)u(x)*] — E[u(x)]E[u(x)"] = cov[u(x)]. (2.300) 


(x) By changing variables using y = z/o, show that the density (2.236) will be 
correctly normalized, provided f(z) is correctly normalized. 


œ» FY Consider a histogram-like density model in which the space x is di- 
vided into fixed regions for which the density p(x) takes the constant value h; over 
the it™ region, and that the volume of region i is denoted A;. Suppose we have a set 
of N observations of x such that n; of these observations fall in region 7. Using a 
Lagrange multiplier to enforce the normalization constraint on the density, derive an 
expression for the maximum likelihood estimator for the {h; }. 


(x) Show that the K’-nearest-neighbour density model defines an improper distribu- 
tion whose integral over all space is divergent. 


Linear 


i 


f m 


~ « 


The focus so far in this book has been on unsupervised learning, including topics 
such as density estimation and data clustering. We turn now to a discussion of super- 
vised learning, starting with regression. The goal of regression is to predict the value 
of one or more continuous target variables t given the value of a D-dimensional vec- 
tor x of input variables. We have already encountered an example of a regression 
problem when we considered polynomial curve fitting in Chapter 1. The polynomial 
is a specific example of a broad class of functions called linear regression models, 
which share the property of being linear functions of the adjustable parameters, and 
which will form the focus of this chapter. The simplest form of linear regression 
models are also linear functions of the input variables. However, we can obtain a 
much more useful class of functions by taking linear combinations of a fixed set of 
nonlinear functions of the input variables, known as basis functions. Such models 
are linear functions of the parameters, which gives them simple analytical properties, 
and yet can be nonlinear with respect to the input variables. 
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3.1. 


Given a training data set comprising N observations {x,,}, where n = 1,..., N, 
together with corresponding target values {tn }, the goal is to predict the value of t 
for a new value of x. In the simplest approach, this can be done by directly con- 
structing an appropriate function y(x) whose values for new inputs x constitute the 
predictions for the corresponding values of t. More generally, from a probabilistic 
perspective, we aim to model the predictive distribution p(t|x) because this expresses 
our uncertainty about the value of t for each value of x. From this conditional dis- 
tribution we can make predictions of t, for any new value of x, in such a way as to 
minimize the expected value of a suitably chosen loss function. As discussed in Sec- 
tion 1.5.5, a common choice of loss function for real-valued variables is the squared 
loss, for which the optimal solution is given by the conditional expectation of t. 

Although linear models have significant limitations as practical techniques for 
pattern recognition, particularly for problems involving input spaces of high dimen- 
sionality, they have nice analytical properties and form the foundation for more so- 
phisticated models to be discussed in later chapters. 


Linear Basis Function Models 


The simplest linear model for regression is one that involves a linear combination of 
the input variables 


y(x, w) = wo twitit...+wprtp (3.1) 


where x = (£1,..., £p)". This is often simply known as linear regression. The key 
property of this model is that it is a linear function of the parameters wo, ..., wp. Itis 
also, however, a linear function of the input variables x;, and this imposes significant 
limitations on the model. We therefore extend the class of models by considering 
linear combinations of fixed nonlinear functions of the input variables, of the form 


M-1 


y(x,w) = wo + > wy; (x) (3.2) 


j=l 


where ¢;(x) are known as basis functions. By denoting the maximum value of the 
index j by M — 1, the total number of parameters in this model will be M. 

The parameter wo allows for any fixed offset in the data and is sometimes called 
a bias parameter (not to be confused with ‘bias’ in a statistical sense). It is often 
convenient to define an additional dummy ‘basis function’ ¢o(x) = 1 so that 


M-1 
y(x, w) = D wjoj(x) = w' (x) (3.3) 
j=0 
where w = (woọ,...,wm-—1)" and @ = (¢o,.-.,@m-—1)". In many practical ap- 


plications of pattern recognition, we will apply some form of fixed pre-processing, 
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or feature extraction, to the original data variables. If the original variables com- 
prise the vector x, then the features can be expressed in terms of the basis functions 
{;(x)}- 

By using nonlinear basis functions, we allow the function y(x, w) to be a non- 
linear function of the input vector x. Functions of the form (3.2) are called linear 
models, however, because this function is linear in w. It is this linearity in the pa- 
rameters that will greatly simplify the analysis of this class of models. However, it 
also leads to some significant limitations, as we discuss in Section 3.6. 

The example of polynomial regression considered in Chapter 1 is a particular 
example of this model in which there is a single input variable x, and the basis func- 
tions take the form of powers of x so that ġ;(x) = x. One limitation of polynomial 
basis functions is that they are global functions of the input variable, so that changes 
in one region of input space affect all other regions. This can be resolved by dividing 
the input space up into regions and fit a different polynomial in each region, leading 
to spline functions (Hastie et al., 2001). 

There are many other possible choices for the basis functions, for example 


oj (x) = exp Se} (3.4) 


28? 


where the jz; govern the locations of the basis functions in input space, and the pa- 

rameter s governs their spatial scale. These are usually referred to as ‘Gaussian’ 

basis functions, although it should be noted that they are not required to have a prob- 

abilistic interpretation, and in particular the normalization coefficient is unimportant 

because these basis functions will be multiplied by adaptive parameters wy. 
Another possibility is the sigmoidal basis function of the form 


a? 
gle) = o (==) 83.5) 
where o(a) is the logistic sigmoid function defined by 
() = 3.6) 
oa 1 + exp(—a) ` 


Equivalently, we can use the ‘tanh’ function because this is related to the logistic 
sigmoid by tanh(a) = 2o(a) — 1, and so a general linear combination of logistic 
sigmoid functions is equivalent to a general linear combination of ‘tanh’ functions. 
These various choices of basis function are illustrated in Figure 3.1. 

Yet another possible choice of basis function is the Fourier basis, which leads to 
an expansion in sinusoidal functions. Each basis function represents a specific fre- 
quency and has infinite spatial extent. By contrast, basis functions that are localized 
to finite regions of input space necessarily comprise a spectrum of different spatial 
frequencies. In many signal processing applications, it is of interest to consider ba- 
sis functions that are localized in both space and frequency, leading to a class of 
functions known as wavelets. These are also defined to be mutually orthogonal, to 
simplify their application. Wavelets are most applicable when the input values live 
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Figure 3.1 Examples of basis functions, showing polynomials on the left, Gaussians of the form (3.4) in the 
centre, and sigmoidal of the form (3.5) on the right. 


Section 1.5.5 


on a regular lattice, such as the successive time points in a temporal sequence, or the 
pixels in an image. Useful texts on wavelets include Ogden (1997), Mallat (1999), 
and Vidakovic (1999). 

Most of the discussion in this chapter, however, is independent of the particular 
choice of basis function set, and so for most of our discussion we shall not specify 
the particular form of the basis functions, except for the purposes of numerical il- 
lustration. Indeed, much of our discussion will be equally applicable to the situation 
in which the vector @(x) of basis functions is simply the identity (x) = x. Fur- 
thermore, in order to keep the notation simple, we shall focus on the case of a single 
target variable t. However, in Section 3.1.5, we consider briefly the modifications 
needed to deal with multiple target variables. 


3.1.1 Maximum likelihood and least squares 


In Chapter 1, we fitted polynomial functions to data sets by minimizing a sum- 
of-squares error function. We also showed that this error function could be motivated 
as the maximum likelihood solution under an assumed Gaussian noise model. Let 
us return to this discussion and consider the least squares approach, and its relation 
to maximum likelihood, in more detail. 

As before, we assume that the target variable t is given by a deterministic func- 
tion y(x, w) with additive Gaussian noise so that 


t = y(x,w) +€ (3.7) 


where € is a zero mean Gaussian random variable with precision (inverse variance) 
B. Thus we can write 


p(t|x, w, 3) = N(tly(x, w), 867+). (3.8) 


Recall that, if we assume a squared loss function, then the optimal prediction, for a 
new value of x, will be given by the conditional mean of the target variable. In the 
case of a Gaussian conditional distribution of the form (3.8), the conditional mean 
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will be simply 


Bit 


x| = [wes dt = y(x,w). (3.9) 


Note that the Gaussian noise assumption implies that the conditional distribution of 
t given x is unimodal, which may be inappropriate for some applications. An ex- 
tension to mixtures of conditional Gaussian distributions, which permit multimodal 
conditional distributions, will be discussed in Section 14.5.1. 

Now consider a data set of inputs X = {x1, . . . , Xy } with corresponding target 
values t;,...,¢,. We group the target variables {t,,} into a column vector that we 
denote by t where the typeface is chosen to distinguish it from a single observation 
of a multivariate target, which would be denoted t. Making the assumption that 
these data points are drawn independently from the distribution (3.8), we obtain the 
following expression for the likelihood function, which is a function of the adjustable 
parameters w and £, in the form 


N 


PIX, w, 8) = | [ Nnw Gn), 87") (3.10) 


n=1 


where we have used (3.3). Note that in supervised learning problems such as regres- 
sion (and classification), we are not seeking to model the distribution of the input 
variables. Thus x will always appear in the set of conditioning variables, and so 
from now on we will drop the explicit x from expressions such as p(t|x, w, 8) in or- 
der to keep the notation uncluttered. Taking the logarithm of the likelihood function, 
and making use of the standard form (1.46) for the univariate Gaussian, we have 


N 
Inp(tlw, b) = Son (talwT (xn), 871) 


n=1 


z Ang 2 + In(2n) En (3.11) 


where the sum-of-squares error function is defined by 


N 
Ep(w) = ; S {tn — W'A(Xn)}. (3.12) 


Having written down the likelihood function, we can use maximum likelihood to 
determine w and (3. Consider first the maximization with respect to w. As observed 
already in Section 1.2.5, we see that maximization of the likelihood function under a 
conditional Gaussian noise distribution for a linear model is equivalent to minimizing 
a sum-of-squares error function given by E'p(w). The gradient of the log likelihood 
function (3.11) takes the form 


N 
Vinp(t}w, 8) = X` {tn — w' (Xn) } bn)". (3.13) 
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Setting this gradient to zero gives 


N N 
0= S 0 tnd(Xn)” -wT B ana) (3.14) 


Solving for w we obtain 
wur = (T8) 'a"t (3.15) 


which are known as the normal equations for the least squares problem. Here ® is an 
NxM matrix, called the design matrix, whose elements are given by ®,,; = $;(Xn), 


so that 
bo(x1) i(Ki) +++ Gu—1(X1) 
Be a i i o (3:16) 
do(xw) 61 (xw) nie dman) 
The quantity 


bi = (Tp) PT (3.17) 
is known as the Moore-Penrose pseudo-inverse of the matrix ® (Rao and Mitra, 
1971; Golub and Van Loan, 1996). It can be regarded as a generalization of the 
notion of matrix inverse to nonsquare matrices. Indeed, if ® is square and invertible, 
then using the property (AB)~! = B~!A~! we see that 6' = 671, 

At this point, we can gain some insight into the role of the bias parameter wọ. If 
we make the bias parameter explicit, then the error function (3.12) becomes 


ee M-1 
Ep(w) = 5 yt — wo — `> w7o;(Xn)}?. (3.18) 
n=1 j=l 


Wo = t = Wi Pj (3.19) 
where we have defined 


ie = LS 
=p} t & == 2 bn) (3.20) 


Thus the bias wọ compensates for the difference between the averages (over the 
training set) of the target values and the weighted sum of the averages of the basis 
function values. 

We can also maximize the log likelihood function (3.11) with respect to the noise 
precision parameter (3, giving 


a -42t — wit P(Xn)}? (3.21) 


3.1. Linear Basis Function Models 143 


Figure 3.2 Geometrical interpretation of the least-squares S 


Exercise 3.2 


solution, in an N-dimensional space whose axes 

are the values of ti,...,¢n. The least-squares t 
regression function is obtained by finding the or- 

thogonal projection of the data vector t onto the 

subspace spanned by the basis functions ¢,; (x) Pı y 

in which each basis function is viewed as a vec- 

tor ~, of length N with elements ¢; (xn). 


and so we see that the inverse of the noise precision is given by the residual variance 
of the target values around the regression function. 


3.1.2 Geometry of least squares 


At this point, it is instructive to consider the geometrical interpretation of the 
least-squares solution. To do this we consider an N-dimensional space whose axes 
are given by the tn, so that t = (t1,... sty) is a vector in this space. Each basis 
function ġ; (Xn ), evaluated at the N data points, can also be represented as a vector in 
the same space, denoted by ẹ;, as illustrated in Figure 3.2. Note that p; corresponds 
to the jt® column of ®, whereas #(x,,) corresponds to the nt® row of ®. If the 
number M of basis functions is smaller than the number N of data points, then the 
M vectors $;(Xn) will span a linear subspace S of dimensionality M. We define 
y to be an N-dimensional vector whose n‘" element is given by y(xn,w), where 
n = 1,..., N. Because y is an arbitrary linear combination of the vectors ~,, it can 
live anywhere in the /-dimensional subspace. The sum-of-squares error (3.12) is 
then equal (up to a factor of 1/2) to the squared Euclidean distance between y and 
t. Thus the least-squares solution for w corresponds to that choice of y that lies in 
subspace S and that is closest to t. Intuitively, from Figure 3.2, we anticipate that 
this solution corresponds to the orthogonal projection of t onto the subspace S. This 
is indeed the case, as can easily be verified by noting that the solution for y is given 
by Pwmu, and then confirming that this takes the form of an orthogonal projection. 

In practice, a direct solution of the normal equations can lead to numerical diffi- 
culties when ®* @ is close to singular. In particular, when two or more of the basis 
vectors p; are co-linear, or nearly so, the resulting parameter values can have large 
magnitudes. Such near degeneracies will not be uncommon when dealing with real 
data sets. The resulting numerical difficulties can be addressed using the technique 
of singular value decomposition, or SVD (Press et al., 1992; Bishop and Nabney, 
2008). Note that the addition of a regularization term ensures that the matrix is non- 
singular, even in the presence of degeneracies. 


3.1.3 Sequential learning 


Batch techniques, such as the maximum likelihood solution (3.15), which in- 
volve processing the entire training set in one go, can be computationally costly for 
large data sets. As we have discussed in Chapter 1, if the data set is sufficiently large, 
it may be worthwhile to use sequential algorithms, also known as on-line algorithms, 
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in which the data points are considered one at a time, and the model parameters up- 
dated after each such presentation. Sequential learning is also appropriate for real- 
time applications in which the data observations are arriving in a continuous stream, 
and predictions must be made before all of the data points are seen. 

We can obtain a sequential learning algorithm by applying the technique of 
stochastic gradient descent, also known as sequential gradient descent, as follows. If 
the error function comprises a sum over data points E = 5°, En, then after presen- 
tation of pattern n, the stochastic gradient descent algorithm updates the parameter 


vector w using 
wt) = w — VE, (3.22) 


where 7 denotes the iteration number, and n is a learning rate parameter. We shall 
discuss the choice of value for 7) shortly. The value of w is initialized to some starting 
vector w(°). For the case of the sum-of-squares error function (3.12), this gives 


where p, = (Xn). This is known as least-mean-squares or the LMS algorithm. 
The value of ņ needs to be chosen with care to ensure that the algorithm converges 
(Bishop and Nabney, 2008). 


3.1.4 Regularized least squares 


In Section 1.1, we introduced the idea of adding a regularization term to an 
error function in order to control over-fitting, so that the total error function to be 
minimized takes the form 

Ep(w) + AEw (w) (3.24) 


where A is the regularization coefficient that controls the relative importance of the 
data-dependent error Æp(w) and the regularization term Æw (w). One of the sim- 
plest forms of regularizer is given by the sum-of-squares of the weight vector ele- 
ments 


Ew(w) = swiw, (3.25) 
If we also consider the sum-of-squares error function given by 
Wals mes 6.26) 
2 n=1 
then the total error function becomes 
Ly — w"b(xn)}? + SwTw. (3.27) 
2 = 2 


This particular choice of regularizer is known in the machine learning literature as 
weight decay because in sequential learning algorithms, it encourages weight values 
to decay towards zero, unless supported by the data. In statistics, it provides an ex- 
ample of a parameter shrinkage method because it shrinks parameter values towards 
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TPPP 


Figure 3.3 Contours of the regularization term in (3.29) for various values of the parameter q. 


Exercise 3.5 


Appendix E 


zero. It has the advantage that the error function remains a quadratic function of 
w, and so its exact minimizer can be found in closed form. Specifically, setting the 
gradient of (3.27) with respect to w to zero, and solving for w as before, we obtain 


w= (1+ 7S) ' 6't. (3.28) 


This represents a simple extension of the least-squares solution (3.15). 
A more general regularizer is sometimes used, for which the regularized error 
takes the form 


De — Ww? b(Xn)}? + 3 Dow (3.29) 


where q = 2 corresponds to the quadratic airs (3.27). Figure 3.3 shows con- 
tours of the regularization function for different values of q. 

The case of q = 1 is know as the lasso in the statistics literature (Tibshirani, 
1996). It has the property that if is sufficiently large, some of the coefficients 
wj are driven to zero, leading to a sparse model in which the corresponding basis 
functions play no role. To see this, we first note that minimizing (3.29) is equivalent 
to minimizing the unregularized sum-of-squares error (3.12) subject to the constraint 


M 
So wil? <n (3.30) 
j=l 


for an appropriate value of the parameter 7, where the two approaches can be related 
using Lagrange multipliers. The origin of the sparsity can be seen from Figure 3.4, 
which shows that the minimum of the error function, subject to the constraint (3.30). 
As À is increased, so an increasing number of parameters are driven to zero. 

Regularization allows complex models to be trained on data sets of limited size 
without severe over-fitting, essentially by limiting the effective model complexity. 
However, the problem of determining the optimal model complexity is then shifted 
from one of finding the appropriate number of basis functions to one of determining 
a suitable value of the regularization coefficient A. We shall return to the issue of 
model complexity later in this chapter. 
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Figure 3.4 Plot of the contours 
of the unregularized error function 
(blue) along with the constraint re- 
gion (3.30) for the quadratic regular- 
izer q = 2 on the left and the lasso 
regularizer q = 1 on the right, in 
which the optimum value for the pa- 
rameter vector w is denoted by w*. 
The lasso gives a sparse solution in 
which wf = 0. 


For the remainder of this chapter we shall focus on the quadratic regularizer 
(3.27) both for its practical importance and its analytical tractability. 


3.1.5 Multiple outputs 


So far, we have considered the case of a single target variable t. In some applica- 
tions, we may wish to predict K > 1 target variables, which we denote collectively 
by the target vector t. This could be done by introducing a different set of basis func- 
tions for each component of t, leading to multiple, independent regression problems. 
However, a more interesting, and more common, approach is to use the same set of 
basis functions to model all of the components of the target vector so that 


y(x,w) = W' d(x) (3.31) 


where y is a k-dimensional column vector, W is an M x K matrix of parameters, 
and @(x) is an M-dimensional column vector with elements ¢;(x), with ¢o(x) = 1 
as before. Suppose we take the conditional distribution of the target vector to be an 
isotropic Gaussian of the form 


p(t|x, W, 8) = N(t|WT d(x), 6D). (3.32) 


If we have a set of observations t;,...,t~, we can combine these into a matrix T 
of size N x K such that the nt row is given by tT. Similarly, we can combine the 
input vectors X1, ..., Xy into a matrix X. The log likelihood function is then given 
by 


N 
Inp(T|X,W,8) = J InN (tp|WT¢(x,), 6I) 


mal 


CR R 
= in( ) 3 2 llên = Wree. (3.33) 
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As before, we can maximize this function with respect to W, giving 
Wan = (818) ` BTT. (3.34) 
If we examine this result for each target variable tg, we have 
wh = (7S) | ST, = Sit, (3.35) 


where t;, is an N-dimensional column vector with components t,,, for n = 1,...N. 
Thus the solution to the regression problem decouples between the different target 
variables, and we need only compute a single pseudo-inverse matrix ®', which is 
shared by all of the vectors wy. 

The extension to general Gaussian noise distributions having arbitrary covari- 
ance matrices is straightforward. Again, this leads to a decoupling into K inde- 
pendent regression problems. This result is unsurprising because the parameters W 
define only the mean of the Gaussian noise distribution, and we know from Sec- 
tion 2.3.4 that the maximum likelihood solution for the mean of a multivariate Gaus- 
sian is independent of the covariance. From now on, we shall therefore consider a 
single target variable t for simplicity. 


The Bias-Variance Decomposition 


So far in our discussion of linear models for regression, we have assumed that the 
form and number of basis functions are both fixed. As we have seen in Chapter 1, 
the use of maximum likelihood, or equivalently least squares, can lead to severe 
over-fitting if complex models are trained using data sets of limited size. However, 
limiting the number of basis functions in order to avoid over-fitting has the side 
effect of limiting the flexibility of the model to capture interesting and important 
trends in the data. Although the introduction of regularization terms can control 
over-fitting for models with many parameters, this raises the question of how to 
determine a suitable value for the regularization coefficient A. Seeking the solution 
that minimizes the regularized error function with respect to both the weight vector 
w and the regularization coefficient À is clearly not the right approach since this 
leads to the unregularized solution with A = 0. 

As we have seen in earlier chapters, the phenomenon of over-fitting is really an 
unfortunate property of maximum likelihood and does not arise when we marginalize 
over parameters in a Bayesian setting. In this chapter, we shall consider the Bayesian 
view of model complexity in some depth. Before doing so, however, it is instructive 
to consider a frequentist viewpoint of the model complexity issue, known as the bias- 
variance trade-off. Although we shall introduce this concept in the context of linear 
basis function models, where it is easy to illustrate the ideas using simple examples, 
the discussion has more general applicability. 

In Section 1.5.5, when we discussed decision theory for regression problems, 
we considered various loss functions each of which leads to a corresponding optimal 
prediction once we are given the conditional distribution p(t|x). A popular choice is 
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the squared loss function, for which the optimal prediction is given by the conditional 
expectation, which we denote by h(x) and which is given by 


h(x) = Eft 


x] = [tan dt. (3.36) 


At this point, it is worth distinguishing between the squared loss function arising 
from decision theory and the sum-of-squares error function that arose in the maxi- 
mum likelihood estimation of model parameters. We might use more sophisticated 
techniques than least squares, for example regularization or a fully Bayesian ap- 
proach, to determine the conditional distribution p(t|x). These can all be combined 
with the squared loss function for the purpose of making predictions. 

We showed in Section 1.5.5 that the expected squared loss can be written in the 
form 


r= f {ue - x) }” p(x) Jat fix ) — t} p(x, t)dxdt. (3.37) 


Recall that the second term, which is independent of y(x), arises from the intrinsic 
noise on the data and represents the minimum achievable value of the expected loss. 
The first term depends on our choice for the function y(x), and we will seek a so- 
lution for y(x) which makes this term a minimum. Because it is nonnegative, the 
smallest that we can hope to make this term is zero. If we had an unlimited supply of 
data (and unlimited computational resources), we could in principle find the regres- 
sion function h(x) to any desired degree of accuracy, and this would represent the 
optimal choice for y(x). However, in practice we have a data set D containing only 
a finite number N of data points, and consequently we do not know the regression 
function h(x) exactly. 

If we model the h(x) using a parametric function y(x, w) governed by a pa- 
rameter vector w, then from a Bayesian perspective the uncertainty in our model is 
expressed through a posterior distribution over w. A frequentist treatment, however, 
involves making a point estimate of w based on the data set D, and tries instead 
to interpret the uncertainty of this estimate through the following thought experi- 
ment. Suppose we had a large number of data sets each of size N and each drawn 
independently from the distribution p(t,x). For any given data set D, we can run 
our learning algorithm and obtain a prediction function y(x; D). Different data sets 
from the ensemble will give different functions and consequently different values of 
the squared loss. The performance of a particular learning algorithm is then assessed 
by taking the average over this ensemble of data sets. 

Consider the integrand of the first term in (3.37), which for a particular data set 
D takes the form 


{y(x;D) — h(x) ¥. (3.38) 


Because this quantity will be dependent on the particular data set D, we take its aver- 
age over the ensemble of data sets. If we add and subtract the quantity Ep[y(x; D)| 
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inside the braces, and then expand, we obtain 


{y(x; D) — Ep[y(x; D)] + Ep[y(x; D)] — h(x) }? 
= {y(x;D) — Eply(x; D) Y + {Ep[y(x; D)] — h(x) }? 
+2{y(x;D) — Ep|y(x; D)|}{Ep[y(x; D)] — h(x)}. (3:39) 


We now take the expectation of this expression with respect to D and note that the 
final term will vanish, giving 


Ep [{y(x; D) — A(x) }7] 


= {Ely(x;D)] - h(x) } + Ep [{y(x:D) — Evly(x;D)}}"] . G40) 
(bias)? variance 


We see that the expected squared difference between y(x;D) and the regression 
function h(x) can be expressed as the sum of two terms. The first term, called the 
squared bias, represents the extent to which the average prediction over all data sets 
differs from the desired regression function. The second term, called the variance, 
measures the extent to which the solutions for individual data sets vary around their 
average, and hence this measures the extent to which the function y(x; D) is sensitive 
to the particular choice of data set. We shall provide some intuition to support these 
definitions shortly when we consider a simple example. 

So far, we have considered a single input value x. If we substitute this expansion 
back into (3.37), we obtain the following decomposition of the expected squared loss 


expected loss = (bias)? + variance + noise (3.41) 


where 


(bias)? = it Ep[y(x;D)] — h(x)}?p(x) dx (3.42) 


variance = f Ep [{y(x;D) — Ep[y(x;D)]}7] p(x) dx (3.43) 
noise = frw — t}°p(x, t) dxdt (3.44) 


and the bias and variance terms now refer to integrated quantities. 

Our goal is to minimize the expected loss, which we have decomposed into the 
sum of a (squared) bias, a variance, and a constant noise term. As we shall see, there 
is a trade-off between bias and variance, with very flexible models having low bias 
and high variance, and relatively rigid models having high bias and low variance. 
The model with the optimal predictive capability is the one that leads to the best 
balance between bias and variance. This is illustrated by considering the sinusoidal 
data set from Chapter 1. Here we generate 100 data sets, each containing N = 25 
data points, independently from the sinusoidal curve h(x) = sin(2rmx). The data 
sets are indexed by | = 1,..., L, where L = 100, and for each data set D we 
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Figure 3.5 Illustration of the dependence of bias and variance on model complexity, governed by a regulariza- 
tion parameter A, using the sinusoidal data set from Chapter 1. There are L = 100 data sets, each having N = 25 
data points, and there are 24 Gaussian basis functions in the model so that the total number of parameters is 
M = 25 including the bias parameter. The left column shows the result of fitting the model to the data sets for 
various values of In A (for clarity, only 20 of the 100 fits are shown). The right column shows the corresponding 
average of the 100 fits (red) along with the sinusoidal function from which the data sets were generated (green). 
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Figure 3.6 Plot of squared bias and variance, 
together with their sum, correspond- 
ing to the results shown in Fig- 
ure 3.5. Also shown is the average 0.12; 


test set error for a test data set size (bias)? + variance 

of 1000 points. The minimum value 9.99 t test error 

of (bias)? + variance occurs around ee 
ln å = —0.31, which is close to the | 


value that gives the minimum error 
on the test data. 


(bias)? 


variance 


fit a model with 24 Gaussian basis functions by minimizing the regularized error 
function (3.27) to give a prediction function y (x) as shown in Figure 3.5. The 
top row corresponds to a large value of the regularization coefficient A that gives low 
variance (because the red curves in the left plot look similar) but high bias (because 
the two curves in the right plot are very different). Conversely on the bottom row, for 
which A is small, there is large variance (shown by the high variability between the 
red curves in the left plot) but low bias (shown by the good fit between the average 
model fit and the original sinusoidal function). Note that the result of averaging many 
solutions for the complex model with M = 25 is a very good fit to the regression 
function, which suggests that averaging may be a beneficial procedure. Indeed, a 
weighted averaging of multiple solutions lies at the heart of a Bayesian approach, 
although the averaging is with respect to the posterior distribution of parameters, not 
with respect to multiple data sets. 

We can also examine the bias-variance trade-off quantitatively for this example. 
The average prediction is estimated from 


L 
(a) = Z 2 y (x) (3.45) 
and the integrated squared bias and integrated variance are then given by 
ooo . Ie > 
(bias)? = > >, a — hlen) (3.46) 
. .. Ee SE _ ge 
variance = N 3 T Ds {y (tn) — U(2n) } (3.47) 


where the integral over x weighted by the distribution p(x) is approximated by a 
finite sum over data points drawn from that distribution. These quantities, along 
with their sum, are plotted as a function of ln À in Figure 3.6. We see that small 
values of A allow the model to become finely tuned to the noise on each individual 


152 


3. LINEAR MODELS FOR REGRESSION 


3.3. 


data set leading to large variance. Conversely, a large value of A pulls the weight 
parameters towards zero leading to large bias. 

Although the bias-variance decomposition may provide some interesting in- 
sights into the model complexity issue from a frequentist perspective, it is of lim- 
ited practical value, because the bias-variance decomposition is based on averages 
with respect to ensembles of data sets, whereas in practice we have only the single 
observed data set. If we had a large number of independent training sets of a given 
size, we would be better off combining them into a single large training set, which 
of course would reduce the level of over-fitting for a given model complexity. 

Given these limitations, we turn in the next section to a Bayesian treatment of 
linear basis function models, which not only provides powerful insights into the 
issues of over-fitting but which also leads to practical techniques for addressing the 
question model complexity. 


Bayesian Linear Regression 


In our discussion of maximum likelihood for setting the parameters of a linear re- 
gression model, we have seen that the effective model complexity, governed by the 
number of basis functions, needs to be controlled according to the size of the data 
set. Adding a regularization term to the log likelihood function means the effective 
model complexity can then be controlled by the value of the regularization coeffi- 
cient, although the choice of the number and form of the basis functions is of course 
still important in determining the overall behaviour of the model. 

This leaves the issue of deciding the appropriate model complexity for the par- 
ticular problem, which cannot be decided simply by maximizing the likelihood func- 
tion, because this always leads to excessively complex models and over-fitting. In- 
dependent hold-out data can be used to determine model complexity, as discussed 
in Section 1.3, but this can be both computationally expensive and wasteful of valu- 
able data. We therefore turn to a Bayesian treatment of linear regression, which will 
avoid the over-fitting problem of maximum likelihood, and which will also lead to 
automatic methods of determining model complexity using the training data alone. 
Again, for simplicity we will focus on the case of a single target variable t. Ex- 
tension to multiple target variables is straightforward and follows the discussion of 
Section 3.1.5. 


3.3.1 Parameter distribution 


We begin our discussion of the Bayesian treatment of linear regression by in- 
troducing a prior probability distribution over the model parameters w. For the mo- 
ment, we shall treat the noise precision parameter ( as a known constant. First note 
that the likelihood function p(t|w) defined by (3.10) is the exponential of a quadratic 
function of w. The corresponding conjugate prior is therefore given by a Gaussian 
distribution of the form 


having mean mo and covariance So. 
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Next we compute the posterior distribution, which is proportional to the product 
of the likelihood function and the prior. Due to the choice of a conjugate Gaus- 
sian prior distribution, the posterior will also be Gaussian. We can evaluate this 
distribution by the usual procedure of completing the square in the exponential, and 
then finding the normalization coefficient using the standard result for a normalized 
Gaussian. However, we have already done the necessary work in deriving the gen- 
eral result (2.116), which allows us to write down the posterior distribution directly 
in the form 


p(w|t) = V(w|my, Sy) (3.49) 

where 
my = Sy (Sp mo + B®"t) (3.50) 
Sy = S+S. (3.51) 


Note that because the posterior distribution is Gaussian, its mode coincides with its 
mean. Thus the maximum posterior weight vector is simply given by wmap = My. 
If we consider an infinitely broad prior Sọ = a` +I with a — 0, the mean my 
of the posterior distribution reduces to the maximum likelihood value wmz; given 
by (3.15). Similarly, if N = 0, then the posterior distribution reverts to the prior. 
Furthermore, if data points arrive sequentially, then the posterior distribution at any 
stage acts as the prior distribution for the subsequent data point, such that the new 
posterior distribution is again given by (3.49). 

For the remainder of this chapter, we shall consider a particular form of Gaus- 
sian prior in order to simplify the treatment. Specifically, we consider a zero-mean 
isotropic Gaussian governed by a single precision parameter a so that 


p(wla) = N(wI0, a7 'T) (3.52) 
and the corresponding posterior distribution over w is then given by (3.49) with 


my = pSye't (3.53) 
Pe al + ETS. (3.54) 


The log of the posterior distribution is given by the sum of the log likelihood and 
the log of the prior and, as a function of w, takes the form 


In p(w|t) = yt —w' (xn)? — sw w + const. (3.55) 


Maximization of this posterior distribution with respect to w is therefore equiva- 
lent to the minimization of the sum-of-squares error function with the addition of a 
quadratic regularization term, corresponding to (3.27) with A = a/{. 

We can illustrate Bayesian learning in a linear basis function model, as well as 
the sequential update of a posterior distribution, using a simple example involving 
straight-line fitting. Consider a single input variable x, a single target variable t and 
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a linear model of the form y(x,w) = wo + wx. Because this has just two adap- 
tive parameters, we can plot the prior and posterior distributions directly in parameter 
space. We generate synthetic data from the function f (x,a) = ao +aıx with param- 
eter values ao = —0.3 and a; = 0.5 by first choosing values of x, from the uniform 
distribution U(«|—1, 1), then evaluating f(x,,, a), and finally adding Gaussian noise 
with standard deviation of 0.2 to obtain the target values t„. Our goal is to recover 
the values of ap and a, from such data, and we will explore the dependence on the 
size of the data set. We assume here that the noise variance is known and hence we 
set the precision parameter to its true value 3 = (1/0.2)? = 25. Similarly, we fix 
the parameter œ to 2.0. We shall shortly discuss strategies for determining a and 
b from the training data. Figure 3.7 shows the results of Bayesian learning in this 
model as the size of the data set is increased and demonstrates the sequential nature 
of Bayesian learning in which the current posterior distribution forms the prior when 
a new data point is observed. It is worth taking time to study this figure in detail as 
it illustrates several important aspects of Bayesian inference. The first row of this 
figure corresponds to the situation before any data points are observed and shows a 
plot of the prior distribution in w space together with six samples of the function 
y(x, w) in which the values of w are drawn from the prior. In the second row, we 
see the situation after observing a single data point. The location (x,t) of the data 
point is shown by a blue circle in the right-hand column. In the left-hand column is a 
plot of the likelihood function p(t|x, w) for this data point as a function of w. Note 
that the likelihood function provides a soft constraint that the line must pass close to 
the data point, where close is determined by the noise precision 3. For comparison, 
the true parameter values a7 = —0.3 and a; = 0.5 used to generate the data set 
are shown by a white cross in the plots in the left column of Figure 3.7. When we 
multiply this likelihood function by the prior from the top row, and normalize, we 
obtain the posterior distribution shown in the middle plot on the second row. Sam- 
ples of the regression function y(x, w) obtained by drawing samples of w from this 
posterior distribution are shown in the right-hand plot. Note that these sample lines 
all pass close to the data point. The third row of this figure shows the effect of ob- 
serving a second data point, again shown by a blue circle in the plot in the right-hand 
column. The corresponding likelihood function for this second data point alone is 
shown in the left plot. When we multiply this likelihood function by the posterior 
distribution from the second row, we obtain the posterior distribution shown in the 
middle plot of the third row. Note that this is exactly the same posterior distribution 
as would be obtained by combining the original prior with the likelihood function 
for the two data points. This posterior has now been influenced by two data points, 
and because two points are sufficient to define a line this already gives a relatively 
compact posterior distribution. Samples from this posterior distribution give rise to 
the functions shown in red in the third column, and we see that these functions pass 
close to both of the data points. The fourth row shows the effect of observing a total 
of 20 data points. The left-hand plot shows the likelihood function for the 20*" data 
point alone, and the middle plot shows the resulting posterior distribution that has 
now absorbed information from all 20 observations. Note how the posterior is much 
sharper than in the third row. In the limit of an infinite number of data points, the 
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likelihood prior/posterior data space 


1 
-1 0 wo ! 


Figure 3.7 Illustration of sequential Bayesian learning for a simple linear model of the form y(x, w) = 
wo + wx. A detailed description of this figure is given in the text. 
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Exercise 3.10 


Exercise 3.11 


posterior distribution would become a delta function centred on the true parameter 
values, shown by the white cross. 

Other forms of prior over the parameters can be considered. For instance, we 
can generalize the Gaussian prior to give 


aie a J at 
p(wla) = É (Z) nial exp (-§ yy ws) (3.56) 


in which q = 2 corresponds to the Gaussian distribution, and only in this case is the 
prior conjugate to the likelihood function (3.10). Finding the maximum of the poste- 
rior distribution over w corresponds to minimization of the regularized error function 
(3.29). In the case of the Gaussian prior, the mode of the posterior distribution was 
equal to the mean, although this will no longer hold if q # 2. 


3.3.2 Predictive distribution 


In practice, we are not usually interested in the value of w itself but rather in 
making predictions of t for new values of x. This requires that we evaluate the 
predictive distribution defined by 


p(t|t, œ, 8) = [ott B)p(wIt, a, 3) dw (3.57) 


in which t is the vector of target values from the training set, and we have omitted the 
corresponding input vectors from the right-hand side of the conditioning statements 
to simplify the notation. The conditional distribution p(t|x, w, 3) of the target vari- 
able is given by (3.8), and the posterior weight distribution is given by (3.49). We 
see that (3.57) involves the convolution of two Gaussian distributions, and so making 
use of the result (2.115) from Section 8.1.4, we see that the predictive distribution 
takes the form 


p(t|x, t, a, 8) = N(thmy (x), on (x) (3.58) 
where the variance 0%, (x) of the predictive distribution is given by 
2 1 T 
on (x) = B + $(x) Sn @(x). (3.59) 


The first term in (3.59) represents the noise on the data whereas the second term 
reflects the uncertainty associated with the parameters w. Because the noise process 
and the distribution of w are independent Gaussians, their variances are additive. 
Note that, as additional data points are observed, the posterior distribution becomes 
narrower. As a consequence it can be shown (Qazaz et al., 1997) that 0%, , (x) < 
ci (x). In the limit N — oo, the second term in (3.59) goes to zero, and the variance 
of the predictive distribution arises solely from the additive noise governed by the 
parameter (. 

As an illustration of the predictive distribution for Bayesian linear regression 
models, let us return to the synthetic sinusoidal data set of Section 1.1. In Figure 3.8, 
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x T 


Figure 3.8 Examples of the predictive distribution (3.58) for a model consisting of 9 Gaussian basis functions 
of the form (3.4) using the synthetic sinusoidal data set of Section 1.1. See the text for a detailed discussion. 


we fit a model comprising a linear combination of Gaussian basis functions to data 
sets of various sizes and then look at the corresponding posterior distributions. Here 
the green curves correspond to the function sin(27x) from which the data points 
were generated (with the addition of Gaussian noise). Data sets of size N = 1, 
N = 2, N = 4, and N = 25 are shown in the four plots by the blue circles. For 
each plot, the red curve shows the mean of the corresponding Gaussian predictive 
distribution, and the red shaded region spans one standard deviation either side of 
the mean. Note that the predictive uncertainty depends on x and is smallest in the 
neighbourhood of the data points. Also note that the level of uncertainty decreases 
as more data points are observed. 

The plots in Figure 3.8 only show the point-wise predictive variance as a func- 
tion of x. In order to gain insight into the covariance between the predictions at 
different values of x, we can draw samples from the posterior distribution over w, 
and then plot the corresponding functions y(x, w), as shown in Figure 3.9. 
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Figure 3.9 Plots of the function y(x, w) using samples from the posterior distributions over w corresponding to 
the plots in Figure 3.8. 


If we used localized basis functions such as Gaussians, then in regions away 
from the basis function centres, the contribution from the second term in the predic- 
tive variance (3.59) will go to zero, leaving only the noise contribution 87+. Thus, 
the model becomes very confident in its predictions when extrapolating outside the 
region occupied by the basis functions, which is generally an undesirable behaviour. 
This problem can be avoided by adopting an alternative Bayesian approach to re- 

Section 6.4 gression known as a Gaussian process. 

Note that, if both w and 8 are treated as unknown, then we can introduce a 

conjugate prior distribution p(w, 8) that, from the discussion in Section 2.3.6, will 
Exercise 3.12 be given by a Gaussian-gamma distribution (Denison et al., 2002). In this case, the 
Exercise 3.13 predictive distribution is a Student’s t-distribution. 


Figure 3.10 The equivalent ker- 

nel k(x, x’) for the Gaussian basis ee eae ee 
functions in Figure 3.1, shown as — 

a plot of x versus 2’, together with 

three slices through this matrix cor- 

responding to three different values 

of x. The data set used to generate 


this kernel comprised 200 values of 
x equally spaced over the interval 


(11. 


Chapter 6 


3.3. Bayesian Linear Regression 159 


3.3.3 Equivalent kernel 


The posterior mean solution (3.53) for the linear basis function model has an in- 
teresting interpretation that will set the stage for kernel methods, including Gaussian 
processes. If we substitute (3.53) into the expression (3.3), we see that the predictive 
mean can be written in the form 


N 
y(x, my) = my P(x) = BO(x)"SyB"t= X BO(K)*SvP(Kn)tn (3-60) 
p=] 


where S y is defined by (3.51). Thus the mean of the predictive distribution at a point 
x is given by a linear combination of the training set target variables tn, so that we 
can write 


y(x,my) = K(X; Xn)tn (3.61) 


where the function 
k(x, x’) = BO(x)"Svo(x’) (3.62) 


is known as the smoother matrix or the equivalent kernel. Regression functions, such 
as this, which make predictions by taking linear combinations of the training set 
target values are known as linear smoothers. Note that the equivalent kernel depends 
on the input values x,, from the data set because these appear in the definition of 
Sy. The equivalent kernel is illustrated for the case of Gaussian basis functions in 
Figure 3.10 in which the kernel functions k(x, x’) have been plotted as a function of 
x’ for three different values of x. We see that they are localized around z, and so the 
mean of the predictive distribution at x, given by y(x, my), is obtained by forming 
a weighted combination of the target values in which data points close to x are given 
higher weight than points further removed from x. Intuitively, it seems reasonable 
that we should weight local evidence more strongly than distant evidence. Note that 
this localization property holds not only for the localized Gaussian basis functions 
but also for the nonlocal polynomial and sigmoidal basis functions, as illustrated in 
Figure 3.11. 
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Figure 3.11 Examples of equiva- 
lent kernels k(x,2’) for x = 0 
plotted as a function of x’, corre- 
sponding (left) to the polynomial ba- 
sis functions and (right) to the sig- 0.02 0.02 
moidal basis functions shown in Fig- 

ure 3.1. Note that these are local- 0 
ized functions of x’ even though the 


0.04 0.04 


corresponding basis functions are 


nonlocal. -1 0 1 -1 (0) 1 


Exercise 3.14 


Further insight into the role of the equivalent kernel can be obtained by consid- 
ering the covariance between y(x) and y(x’), which is given by 


cov[y(x), y(x] = covlp(x)"w, w" d(x’) 
= o(x)"Sv G(x’) =8k(x,x!) (3.63) 


where we have made use of (3.49) and (3.62). From the form of the equivalent 
kernel, we see that the predictive mean at nearby points will be highly correlated, 
whereas for more distant pairs of points the correlation will be smaller. 

The predictive distribution shown in Figure 3.8 allows us to visualize the point- 
wise uncertainty in the predictions, governed by (3.59). However, by drawing sam- 
ples from the posterior distribution over w, and plotting the corresponding model 
functions y(x,w) as in Figure 3.9, we are visualizing the joint uncertainty in the 
posterior distribution between the y values at two (or more) x values, as governed by 
the equivalent kernel. 

The formulation of linear regression in terms of a kernel function suggests an 
alternative approach to regression as follows. Instead of introducing a set of basis 
functions, which implicitly determines an equivalent kernel, we can instead define 
a localized kernel directly and use this to make predictions for new input vectors x, 
given the observed training set. This leads to a practical framework for regression 
(and classification) called Gaussian processes, which will be discussed in detail in 
Section 6.4. 

We have seen that the effective kernel defines the weights by which the training 
set target values are combined in order to make a prediction at a new value of x, and 
it can be shown that these weights sum to one, in other words 


N 


S k(x, Xn) = 1 (3.64) 


n=1 


for all values of x. This intuitively pleasing result can easily be proven informally 
by noting that the summation is equivalent to considering the predictive mean (x) 
for a set of target data in which t,, = 1 for all n. Provided the basis functions are 
linearly independent, that there are more data points than basis functions, and that 
one of the basis functions is constant (corresponding to the bias parameter), then it is 
clear that we can fit the training data exactly and hence that the predictive mean will 
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be simply y(x) = 1, from which we obtain (3.64). Note that the kernel function can 
be negative as well as positive, so although it satisfies a summation constraint, the 
corresponding predictions are not necessarily convex combinations of the training 
set target variables. 

Finally, we note that the equivalent kernel (3.62) satisfies an important property 
shared by kernel functions in general, namely that it can be expressed in the form an 
inner product with respect to a vector ¢(x) of nonlinear functions, so that 


k(x, z) = a(x)" a(z) (3.65) 


where a(x) = B28? Q(x). 


Bayesian Model Comparison 


In Chapter 1, we highlighted the problem of over-fitting as well as the use of cross- 
validation as a technique for setting the values of regularization parameters or for 
choosing between alternative models. Here we consider the problem of model se- 
lection from a Bayesian perspective. In this section, our discussion will be very 
general, and then in Section 3.5 we shall see how these ideas can be applied to the 
determination of regularization parameters in linear regression. 

As we shall see, the over-fitting associated with maximum likelihood can be 
avoided by marginalizing (summing or integrating) over the model parameters in- 
stead of making point estimates of their values. Models can then be compared di- 
rectly on the training data, without the need for a validation set. This allows all 
available data to be used for training and avoids the multiple training runs for each 
model associated with cross-validation. It also allows multiple complexity parame- 
ters to be determined simultaneously as part of the training process. For example, 
in Chapter 7 we shall introduce the relevance vector machine, which is a Bayesian 
model having one complexity parameter for every training data point. 

The Bayesian view of model comparison simply involves the use of probabilities 
to represent uncertainty in the choice of model, along with a consistent application 
of the sum and product rules of probability. Suppose we wish to compare a set of L 
models {M;} where i = 1,..., L. Here a model refers to a probability distribution 
over the observed data D. In the case of the polynomial curve-fitting problem, the 
distribution is defined over the set of target values t, while the set of input values X 
is assumed to be known. Other types of model define a joint distributions over X 
and t. We shall suppose that the data is generated from one of these models but we 
are uncertain which one. Our uncertainty is expressed through a prior probability 
distribution p(M;). Given a training set D, we then wish to evaluate the posterior 
distribution 


p(M:lD) x p(M;i)p(D|M;). (3.66) 


The prior allows us to express a preference for different models. Let us simply 
assume that all models are given equal prior probability. The interesting term is 
the model evidence p(D|M;) which expresses the preference shown by the data for 
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different models, and we shall examine this term in more detail shortly. The model 
evidence is sometimes also called the marginal likelihood because it can be viewed 
as a likelihood function over the space of models, in which the parameters have been 
marginalized out. The ratio of model evidences p(D|M,;)/p(D|M,) for two models 
is known as a Bayes factor (Kass and Raftery, 1995). 

Once we know the posterior distribution over models, the predictive distribution 
is given, from the sum and product rules, by 


L 
p(t|x,D) = $ p(t|x, Mi,D)p(MiID). (3.67) 


=] 


This is an example of a mixture distribution in which the overall predictive distribu- 
tion is obtained by averaging the predictive distributions p(t|x, M;, D) of individual 
models, weighted by the posterior probabilities p(M;|D) of those models. For in- 
stance, if we have two models that are a-posteriori equally likely and one predicts 
a narrow distribution around t = a while the other predicts a narrow distribution 
around t = b, the overall predictive distribution will be a bimodal distribution with 
modes at t = a and t = b, not a single model at t = (a + 6)/2. 

A simple approximation to model averaging is to use the single most probable 
model alone to make predictions. This is known as model selection. 

For a model governed by a set of parameters w, the model evidence is given, 
from the sum and product rules of probability, by 


p(DIM,) = / p(DIw, M,)p(w|M,) dw. (3.68) 


From a sampling perspective, the marginal likelihood can be viewed as the proba- 
bility of generating the data set D from a model whose parameters are sampled at 
random from the prior. It is also interesting to note that the evidence is precisely the 
normalizing term that appears in the denominator in Bayes’ theorem when evaluating 
the posterior distribution over parameters because 


_ p(D|w, Mi)p(w|M;:) 
p(D|M;) l 


p(w|D, Mi) (3.69) 

We can obtain some insight into the model evidence by making a simple approx- 
imation to the integral over parameters. Consider first the case of a model having a 
single parameter w. The posterior distribution over parameters is proportional to 
p(D|w)p(w), where we omit the dependence on the model M; to keep the notation 
uncluttered. If we assume that the posterior distribution is sharply peaked around the 
most probable value wap, with width Awyosterior, then we can approximate the in- 
tegral by the value of the integrand at its maximum times the width of the peak. If we 
further assume that the prior is flat with width Aw,,ior so that p(w) = 1/Awprior, 
then we have 


Aw osterior 
p(D) = f p(D|w)plw)dw ~ p(Dlwmar) mar (3.70) 
prior 
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Figure 3.12 We can obtain a rough approximation to AW posterior 


Section 4.4.1 


the model evidence if we assume that —_— 
the posterior distribution over parame- 

ters is sharply peaked around its mode 

WMAP. 


WMAP w 
a a 
AWprior 
and so taking logs we obtain 
A osterior 
Inp(D) ~ Inp(D|wmap) + In ( sei ) , (3.71) 
AwWprior 


This approximation is illustrated in Figure 3.12. The first term represents the fit to 
the data given by the most probable parameter values, and for a flat prior this would 
correspond to the log likelihood. The second term penalizes the model according to 
its complexity. Because Awposterior < AWprior this term is negative, and it increases 
in magnitude as the ratio AWposterior/AWprior gets smaller. Thus, if parameters are 
finely tuned to the data in the posterior distribution, then the penalty term is large. 

For a model having a set of M parameters, we can make a similar approximation 
for each parameter in turn. Assuming that all parameters have the same ratio of 
AUWposterior / AWprior» we obtain 


(3.72) 


A osterior 
Inp(D) ~ In p(D|wmar) + MIn (>) 


AUWprior 


Thus, in this very simple approximation, the size of the complexity penalty increases 
linearly with the number M of adaptive parameters in the model. As we increase 
the complexity of the model, the first term will typically decrease, because a more 
complex model is better able to fit the data, whereas the second term will increase 
due to the dependence on M. The optimal model complexity, as determined by 
the maximum evidence, will be given by a trade-off between these two competing 
terms. We shall later develop a more refined version of this approximation, based on 
a Gaussian approximation to the posterior distribution. 

We can gain further insight into Bayesian model comparison and understand 
how the marginal likelihood can favour models of intermediate complexity by con- 
sidering Figure 3.13. Here the horizontal axis is a one-dimensional representation 
of the space of possible data sets, so that each point on this axis corresponds to a 
specific data set. We now consider three models M1, Ma and Mg of successively 
increasing complexity. Imagine running these models generatively to produce exam- 
ple data sets, and then looking at the distribution of data sets that result. Any given 
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Figure 3.13 Schematic illustration of the 
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distribution of data sets for p(D) 
three models of different com- 
plexity, in which Mj, is the 
simplest and M3 is the most 
complex. Note that the dis- 
tributions are normalized. In 
this example, for the partic- 
ular observed data set Do, 
the model M2 with intermedi- 
ate complexity has the largest 
evidence. 


model can generate a variety of different data sets since the parameters are governed 
by a prior probability distribution, and for any choice of the parameters there may 
be random noise on the target variables. To generate a particular data set from a spe- 
cific model, we first choose the values of the parameters from their prior distribution 
p(w), and then for these parameter values we sample the data from p(D|w). A sim- 
ple model (for example, based on a first order polynomial) has little variability and 
so will generate data sets that are fairly similar to each other. Its distribution p(D) 
is therefore confined to a relatively small region of the horizontal axis. By contrast, 
a complex model (such as a ninth order polynomial) can generate a great variety of 
different data sets, and so its distribution p(D) is spread over a large region of the 
space of data sets. Because the distributions p(D|M;) are normalized, we see that 
the particular data set Dp can have the highest value of the evidence for the model 
of intermediate complexity. Essentially, the simpler model cannot fit the data well, 
whereas the more complex model spreads its predictive probability over too broad a 
range of data sets and so assigns relatively small probability to any one of them. 

Implicit in the Bayesian model comparison framework is the assumption that 
the true distribution from which the data are generated is contained within the set of 
models under consideration. Provided this is so, we can show that Bayesian model 
comparison will on average favour the correct model. To see this, consider two 
models Mı and Mo, in which the truth corresponds to M,. For a given finite data 
set, itis possible for the Bayes factor to be larger for the incorrect model. However, if 
we average the Bayes factor over the distribution of data sets, we obtain the expected 
Bayes factor in the form 


p(D| M1) 
fromm DDIM) dD (3.73) 


where the average has been taken with respect to the true distribution of the data. 
This quantity is an example of the Kullback-Leibler divergence and satisfies the prop- 
erty of always being positive unless the two distributions are equal in which case it 
is zero. Thus on average the Bayes factor will always favour the correct model. 

We have seen that the Bayesian framework avoids the problem of over-fitting 
and allows models to be compared on the basis of the training data alone. However, 
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a Bayesian approach, like any approach to pattern recognition, needs to make as- 
sumptions about the form of the model, and if these are invalid then the results can 
be misleading. In particular, we see from Figure 3.12 that the model evidence can 
be sensitive to many aspects of the prior, such as the behaviour in the tails. Indeed, 
the evidence is not defined if the prior is improper, as can be seen by noting that 
an improper prior has an arbitrary scaling factor (in other words, the normalization 
coefficient is not defined because the distribution cannot be normalized). If we con- 
sider a proper prior and then take a suitable limit in order to obtain an improper prior 
(for example, a Gaussian prior in which we take the limit of infinite variance) then 
the evidence will go to zero, as can be seen from (3.70) and Figure 3.12. It may, 
however, be possible to consider the evidence ratio between two models first and 
then take a limit to obtain a meaningful answer. 

In a practical application, therefore, it will be wise to keep aside an independent 
test set of data on which to evaluate the overall performance of the final system. 


The Evidence Approximation 


In a fully Bayesian treatment of the linear basis function model, we would intro- 
duce prior distributions over the hyperparameters a and 8 and make predictions by 
marginalizing with respect to these hyperparameters as well as with respect to the 
parameters w. However, although we can integrate analytically over either w or 
over the hyperparameters, the complete marginalization over all of these variables 
is analytically intractable. Here we discuss an approximation in which we set the 
hyperparameters to specific values determined by maximizing the marginal likeli- 
hood function obtained by first integrating over the parameters w. This framework 
is known in the statistics literature as empirical Bayes (Bernardo and Smith, 1994; 
Gelman et al., 2004), or type 2 maximum likelihood (Berger, 1985), or generalized 
maximum likelihood (Wahba, 1975), and in the machine learning literature is also 
called the evidence approximation (Gull, 1989; MacKay, 1992a). 

If we introduce hyperpriors over œ and p, the predictive distribution is obtained 
by marginalizing over w, a and 8 so that 


p(i|t) = f I I p(t|w,8)p(wlt,a, Aawa G7 


where p(t|w, (3) is given by (3.8) and p(w|t, a, 8) is given by (3.49) with my and 
Sy defined by (3.53) and (3.54) respectively. Here we have omitted the dependence 
on the input variable x to keep the notation uncluttered. If the posterior distribution 
p(a, B|t) is sharply peaked around values & and ĝ, then the predictive distribution is 
obtained simply by marginalizing over w in which a and 6 are fixed to the values a 
and (3, so that 


p(t|t) ~ p(t|t, &, 8) = J p(t|w, B)p(wit, a, 3) dw. (3.75) 
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Exercise 3.17 


From Bayes’ theorem, the posterior distribution for a and ( is given by 


pla, Blt) x p(tla, 2)p(a, B). (3.76) 


If the prior is relatively flat, then in the evidence framework the values of Q and 
b are obtained by maximizing the marginal likelihood function p(t|a, 3). We shall 
proceed by evaluating the marginal likelihood for the linear basis function model and 
then finding its maxima. This will allow us to determine values for these hyperpa- 
rameters from the training data alone, without recourse to cross-validation. Recall 
that the ratio a/@ is analogous to a regularization parameter. 

As an aside it is worth noting that, if we define conjugate (Gamma) prior distri- 
butions over a and £, then the marginalization over these hyperparameters in (3.74) 
can be performed analytically to give a Student’s t-distribution over w (see Sec- 
tion 2.3.7). Although the resulting integral over w is no longer analytically tractable, 
it might be thought that approximating this integral, for example using the Laplace 
approximation discussed (Section 4.4) which is based on a local Gaussian approxi- 
mation centred on the mode of the posterior distribution, might provide a practical 
alternative to the evidence framework (Buntine and Weigend, 1991). However, the 
integrand as a function of w typically has a strongly skewed mode so that the Laplace 
approximation fails to capture the bulk of the probability mass, leading to poorer re- 
sults than those obtained by maximizing the evidence (MacKay, 1999). 

Returning to the evidence framework, we note that there are two approaches that 
we can take to the maximization of the log evidence. We can evaluate the evidence 
function analytically and then set its derivative equal to zero to obtain re-estimation 
equations for œ and (3, which we shall do in Section 3.5.2. Alternatively we use a 
technique called the expectation maximization (EM) algorithm, which will be dis- 
cussed in Section 9.3.4 where we shall also show that these two approaches converge 
to the same solution. 


3.5.1 Evaluation of the evidence function 


The marginal likelihood function p(t|a, 3) is obtained by integrating over the 
weight parameters w, so that 


(tla, 8) = / (tlw, 8)p(wla) dw. (3.77) 


One way to evaluate this integral is to make use once again of the result (2.115) 
for the conditional distribution in a linear-Gaussian model. Here we shall evaluate 
the integral instead by completing the square in the exponent and making use of the 
standard form for the normalization coefficient of a Gaussian. 

From (3.11), (3.12), and (3.52), we can write the evidence function in the form 


N/2 r 
tla) = (2) ee [eve aw ew 
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where M is the dimensionality of w, and we have defined 


= eit w|? 4 Swi. (3.79) 


We recognize (3.79) as being equal, up to a constant of proportionality, to the reg- 
ularized sum-of-squares error function (3.27). We now complete the square over w 
giving 


E(w) = E(my) + sw — my)" A(w — my) (3.80) 
where we have introduced 
A=al+6®'® (3.81) 
together with 
E(my) = : |t- my + Smýmy. (3.82) 


Note that A corresponds to the matrix of second derivatives of the error function 
A = VVE(w) (3.83) 
and is known as the Hessian matrix. Here we have also defined my given by 
my = bA Pt. (3.84) 


Using (3.54), we see that A = Sy. and hence (3.84) is equivalent to the previous 
definition (3.53), and therefore represents the mean of the posterior distribution. 

The integral over w can now be evaluated simply by appealing to the standard 
result for the normalization coefficient of a multivariate Gaussian, giving 


J view} aw 


= exp{—E(my)} fo {50 —my)'A(w — my) } dw 
= exp{-E(my) (2r) VP] A]. (3.85) 


Using (3.78) we can then write the log of the marginal likelihood in the form 


M N 1 N 
ln p(tla, 8) = 3 Ine4 5 In@— E(my) — z PIA — z 2n) (3.86) 


which is the required expression for the evidence function. 

Returning to the polynomial regression problem, we can plot the model evidence 
against the order of the polynomial, as shown in Figure 3.14. Here we have assumed 
a prior of the form (1.65) with the parameter a fixed at a = 5 x 1078. The form 
of this plot is very instructive. Referring back to Figure 1.4, we see that the M = 0 
polynomial has very poor fit to the data and consequently gives a relatively low value 
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Figure 3.14 Plot of the model evidence versus 
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the order M, for the polynomial re- —1g + J 
gression model, showing that the 
evidence favours the model with 
M =3. -20f 

-22 p 

-24 

-26 ; 
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for the evidence. Going to the M = 1 polynomial greatly improves the data fit, and 
hence the evidence is significantly higher. However, in going to M = 2, the data 
fit is improved only very marginally, due to the fact that the underlying sinusoidal 
function from which the data is generated is an odd function and so has no even terms 
in a polynomial expansion. Indeed, Figure 1.5 shows that the residual data error is 
reduced only slightly in going from M = 1 to M = 2. Because this richer model 
suffers a greater complexity penalty, the evidence actually falls in going from M = 1 
to M = 2. When we go to M = 3 we obtain a significant further improvement in 
data fit, as seen in Figure 1.4, and so the evidence is increased again, giving the 
highest overall evidence for any of the polynomials. Further increases in the value 
of M produce only small improvements in the fit to the data but suffer increasing 
complexity penalty, leading overall to a decrease in the evidence values. Looking 
again at Figure 1.5, we see that the generalization error is roughly constant between 
M = 3 and M = 8, and it would be difficult to choose between these models on 
the basis of this plot alone. The evidence values, however, show a clear preference 
for M = 3, since this is the simplest model which gives a good explanation for the 
observed data. 


3.5.2 Maximizing the evidence function 


Let us first consider the maximization of p(t|a, 3) with respect to a. This can 
be done by first defining the following eigenvector equation 


(86"®) u; = Ajuj. (3.87) 


From (3.81), it then follows that A has eigenvalues aœ + A;. Now consider the deriva- 
tive of the term involving In |A| in a with respect to a. We have 


d 1 
g lAs io TOs + a) Ja Dent a) = aa (3.88) 
Thus the stationary points of (3.86) with respect to a satisfy 
M 1 1 1 
0=— - m} 3.89 
2a 2 NN 72L Jta ee) 


Exercise 3.20 


Exercise 3.22 
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Multiplying through by 2a and rearranging, we obtain 


oamimy =M—a D =y. (3.90) 


Aj ta 


Since there are M terms in the sum over i, the quantity ~y can be written 


ri 
= : 3.91 
y D T (3.91) 


The interpretation of the quantity y will be discussed shortly. From (3.90) we see 
that the value of a that maximizes the marginal likelihood satisfies 


Y 
T : 
m)My 


a= (3.92) 
Note that this is an implicit solution for a not only because y depends on a, but also 
because the mode my of the posterior distribution itself depends on the choice of 
a. We therefore adopt an iterative procedure in which we make an initial choice for 
q and use this to find my, which is given by (3.53), and also to evaluate y, which 
is given by (3.91). These values are then used to re-estimate œ using (3.92), and the 
process repeated until convergence. Note that because the matrix PT® is fixed, we 
can compute its eigenvalues once at the start and then simply multiply these by ( to 
obtain the \;. 

It should be emphasized that the value of a has been determined purely by look- 
ing at the training data. In contrast to maximum likelihood methods, no independent 
data set is required in order to optimize the model complexity. 

We can similarly maximize the log marginal likelihood (3.86) with respect to 8. 
To do this, we note that the eigenvalues \; defined by (3.87) are proportional to 3, 
and hence dd; /d3 = ;/ 6 giving 


d od ee M 7 
ag gl Oe) Jna 7 (3.93) 


The stationary point of the marginal likelihood therefore satisfies 


2 7 
28 2 D {tn —my (Xn) } — 38 (3.94) 
and rearranging we obtain 
1 b : 2 
B N- do {tn mR). (3.95) 
Y n=1 


Again, this is an implicit solution for 8 and can be solved by choosing an initial 
value for 3 and then using this to calculate my and y and then re-estimate 8 using 
(3.95), repeating until convergence. If both a and 8 are to be determined from the 
data, then their values can be re-estimated together after each update of y. 
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Figure 3.15 Contours of the likelihood function (red) w2 

and the prior (green) in which the axes in parameter 

space have been rotated to align with the eigenvectors 

u; of the Hessian. For a = 0, the mode of the poste- u2 

rior is given by the maximum likelihood solution wm, 

whereas for nonzero a the mode is at wmar = mw. In 

the direction wı the eigenvalue A, defined by (3.87), is uy, 


small compared with a and so the quantity \1/(A1 + a) 
is close to zero, and the corresponding MAP value of 


e. 
WMAP 


wi is also close to zero. By contrast, in the direction w2 
the eigenvalue A2 is large compared with a and so the 
quantity A2/(A2+.«) is close to unity, and the MAP value 
of w2 is close to its maximum likelihood value. 


WI 


3.5.3 Effective number of parameters 


The result (3.92) has an elegant interpretation (MacKay, 1992a), which provides 
insight into the Bayesian solution for a. To see this, consider the contours of the like- 
lihood function and the prior as illustrated in Figure 3.15. Here we have implicitly 
transformed to a rotated set of axes in parameter space aligned with the eigenvec- 
tors u; defined in (3.87). Contours of the likelihood function are then axis-aligned 
ellipses. The eigenvalues A; measure the curvature of the likelihood function, and 
so in Figure 3.15 the eigenvalue 1 is small compared with 2 (because a smaller 
curvature corresponds to a greater elongation of the contours of the likelihood func- 
tion). Because 3®"' @ is a positive definite matrix, it will have positive eigenvalues, 
and so the ratio \;/(A; + a) will lie between 0 and 1. Consequently, the quantity y 
defined by (3.91) will lie in the range 0 < y < M. For directions in which à; > a, 
the corresponding parameter w; will be close to its maximum likelihood value, and 
the ratio A; /(A; + a) will be close to 1. Such parameters are called well determined 
because their values are tightly constrained by the data. Conversely, for directions 
in which A; < a, the corresponding parameters w; will be close to zero, as will the 
ratios A; /(A; +a). These are directions in which the likelihood function is relatively 
insensitive to the parameter value and so the parameter has been set to a small value 
by the prior. The quantity ~y defined by (3.91) therefore measures the effective total 
number of well determined parameters. 

We can obtain some insight into the result (3.95) for re-estimating 3 by com- 
paring it with the corresponding maximum likelihood result given by (3.21). Both 
of these formulae express the variance (the inverse precision) as an average of the 
squared differences between the targets and the model predictions. However, they 
differ in that the number of data points NV in the denominator of the maximum like- 
lihood result is replaced by N — y in the Bayesian result. We recall from (1.56) that 
the maximum likelihood estimate of the variance for a Gaussian distribution over a 
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single variable x is given by 


OML = N DO — pm)? (3.96) 


and that this estimate is biased because the maximum likelihood solution umi for 
the mean has fitted some of the noise on the data. In effect, this has used up one 
degree of freedom in the model. The corresponding unbiased estimate is given by 
(1.59) and takes the form 


OMAP = NI Ye, — pn)? . (3.97) 


We shall see in Section 10.1.3 that this result can be obtained from a Bayesian treat- 
ment in which we marginalize over the unknown mean. The factor of N — 1 in the 
denominator of the Bayesian result takes account of the fact that one degree of free- 
dom has been used in fitting the mean and removes the bias of maximum likelihood. 
Now consider the corresponding results for the linear regression model. The mean 
of the target distribution is now given by the function w' #(x), which contains M 
parameters. However, not all of these parameters are tuned to the data. The effective 
number of parameters that are determined by the data is y, with the remaining M — y 
parameters set to small values by the prior. This is reflected in the Bayesian result 
for the variance that has a factor N — y in the denominator, thereby correcting for 
the bias of the maximum likelihood result. 

We can illustrate the evidence framework for setting hyperparameters using the 
sinusoidal synthetic data set from Section 1.1, together with the Gaussian basis func- 
tion model comprising 9 basis functions, so that the total number of parameters in 
the model is given by M = 10 including the bias. Here, for simplicity of illustra- 
tion, we have set 8 to its true value of 11.1 and then used the evidence framework to 
determine a, as shown in Figure 3.16. 

We can also see how the parameter a controls the magnitude of the parameters 
{w;}, by plotting the individual parameters versus the effective number ~y of param- 
eters, as shown in Figure 3.17. 

If we consider the limit N >> M in which the number of data points is large in 
relation to the number of parameters, then from (3.87) all of the parameters will be 
well determined by the data because &' ® involves an implicit sum over data points, 
and so the eigenvalues A; increase with the size of the data set. In this case, y = M, 
and the re-estimation equations for a and 3 become 


M 
* T ZEw(mx) = 
N 
ee 3.99 
b Daim) (3.99) 


where Ew and Ep are defined by (3.25) and (3.26), respectively. These results 
can be used as an easy-to-compute approximation to the full evidence re-estimation 
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Figure 3.16 The left plot shows y (red curve) and 2aEw (mvn) (blue curve) versus Ina for the sinusoidal 
synthetic data set. It is the intersection of these two curves that defines the optimum value for a given by the 
evidence procedure. The right plot shows the corresponding graph of log evidence In p(t|a, 8) versus In a (red 
curve) showing that the peak coincides with the crossing point of the curves in the left plot. Also shown is the 
test set error (blue curve) showing that the evidence maximum occurs close to the point of best generalization. 


formulae, because they do not require evaluation of the eigenvalue spectrum of the 


Hessian. 

Figure 3.17 Plot of the 10 parameters wi i i i 
from the Gaussian basis function o 
model versus the effective num- 2} A 
ber of parameters ~y, in which the w; , 
hyperparameter a is varied in the it 
range 0 < a < œ causing y to 7 
vary in the range 0 < y < M. 2 

Or 6 
3 
-{ + 1 
© 
—2+ 9 
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3.6. Limitations of Fixed Basis Functions 


Throughout this chapter, we have focussed on models comprising a linear combina- 
tion of fixed, nonlinear basis functions. We have seen that the assumption of linearity 
in the parameters led to a range of useful properties including closed-form solutions 
to the least-squares problem, as well as a tractable Bayesian treatment. Furthermore, 
for a suitable choice of basis functions, we can model arbitrary nonlinearities in the 
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mapping from input variables to targets. In the next chapter, we shall study an anal- 
ogous class of models for classification. 

It might appear, therefore, that such linear models constitute a general purpose 
framework for solving problems in pattern recognition. Unfortunately, there are 
some significant shortcomings with linear models, which will cause us to turn in 
later chapters to more complex models such as support vector machines and neural 
networks. 

The difficulty stems from the assumption that the basis functions @; (x) are fixed 
before the training data set is observed and is a manifestation of the curse of dimen- 
sionality discussed in Section 1.4. As a consequence, the number of basis functions 
needs to grow rapidly, often exponentially, with the dimensionality D of the input 
space. 

Fortunately, there are two properties of real data sets that we can exploit to help 
alleviate this problem. First of all, the data vectors {Xn } typically lie close to a non- 
linear manifold whose intrinsic dimensionality is smaller than that of the input space 
as a result of strong correlations between the input variables. We will see an example 
of this when we consider images of handwritten digits in Chapter 12. If we are using 
localized basis functions, we can arrange that they are scattered in input space only 
in regions containing data. This approach is used in radial basis function networks 
and also in support vector and relevance vector machines. Neural network models, 
which use adaptive basis functions having sigmoidal nonlinearities, can adapt the 
parameters so that the regions of input space over which the basis functions vary 
corresponds to the data manifold. The second property is that target variables may 
have significant dependence on only a small number of possible directions within the 
data manifold. Neural networks can exploit this property by choosing the directions 
in input space to which the basis functions respond. 


(x) FQ Show that the ‘tanh’ function and the logistic sigmoid function (3.6) 
are related by 
tanh(a) = 20(2a) — 1. (3.100) 


Hence show that a general linear combination of logistic sigmoid functions of the 
form 


M 
L— hj 
y(x,w) = wo + 2, wjo (=) (3.101) 
j=l 
is equivalent to a linear combination of ‘tanh’ functions of the form 
M B 
y(x, u) = uo + i tanh ( z 1) (3.102) 
g=l 
and find expressions to relate the new parameters {u,,..., um } to the original pa- 


rameters {w1,..., was}. 
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3.2 


3.3 


3.4 


3.5 


3.6 


(xx) Show that the matrix 
p(T) st (3.103) 


takes any vector v and projects it onto the space spanned by the columns of ®. Use 
this result to show that the least-squares solution (3.15) corresponds to an orthogonal 
projection of the vector t onto the manifold S as shown in Figure 3.2. 


(x) Consider a data set in which each data point t,, is associated with a weighting 
factor r,, > 0, so that the sum-of-squares error function becomes 


N 
Ep(w) = z XO rn {tn — WT P(%n)} . (3.104) 
n=1 


Find an expression for the solution w* that minimizes this error function. Give two 
alternative interpretations of the weighted sum-of-squares error function in terms of 
(i) data dependent noise variance and (ii) replicated data points. 


(x) EQ Consider a linear model of the form 


D 
ylz, w) = wo +) wizi (3.105) 


i=1 


together with a sum-of-squares error function of the form 
La 
Ep(w) = 5 >, {y(tn,W) —ty}’. (3.106) 


Now suppose that Gaussian noise e; with zero mean and variance o° is added in- 
dependently to each of the input variables z;. By making use of Efe;] = 0 and 
Efeicj] = ôij o”, show that minimizing Ep averaged over the noise distribution is 
equivalent to minimizing the sum-of-squares error for noise-free input variables with 
the addition of a weight-decay regularization term, in which the bias parameter wọ 
is omitted from the regularizer. 


(x) FY Using the technique of Lagrange multipliers, discussed in Appendix E, 
show that minimization of the regularized error function (3.29) is equivalent to mini- 
mizing the unregularized sum-of-squares error (3.12) subject to the constraint (3.30). 
Discuss the relationship between the parameters 77 and À. 


(x) EQ Consider a linear basis function regression model for a multivariate 
target variable t having a Gaussian distribution of the form 


p(t|W, =) = N(tly(x, W), 5) (3.107) 


where 
y(x, W) = W' (x) (3.108) 


3.7 


3.8 


3.9 


3.10 


3.11 


3.12 
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together with a training data set comprising input basis vectors @(x,,) and corre- 
sponding target vectors tn, with n = 1,..., N. Show that the maximum likelihood 
solution Ww for the parameter matrix W has the property that each column is 
given by an expression of the form (3.15), which was the solution for an isotropic 
noise distribution. Note that this is independent of the covariance matrix ©. Show 
that the maximum likelihood solution for © is given by 


N 
1 
OT >, (tn — WELO(Xn)) (tn — Win P(Xn)) - (3.109) 


(x) By using the technique of completing the square, verify the result (3.49) for the 
posterior distribution of the parameters w in the linear basis function model in which 
my and S y are defined by (3.50) and (3.51) respectively. 


œ» QTY Consider the linear basis function model in Section 3.1, and suppose 
that we have already observed N data points, so that the posterior distribution over 
w is given by (3.49). This posterior can be regarded as the prior for the next obser- 
vation. By considering an additional data point (xv +1,tN4+1), and by completing 
the square in the exponential, show that the resulting posterior distribution is again 
given by (3.49) but with S y replaced by Sy41 and my replaced by my 414. 


(xx) Repeat the previous exercise but instead of completing the square by hand, 
make use of the general result for linear-Gaussian models given by (2.116). 


œ» [QTY By making use of the result (2.115) to evaluate the integral in (3.57), 
verify that the predictive distribution for the Bayesian linear regression model is 
given by (3.58) in which the input-dependent variance is given by (3.59). 


(xx) We have seen that, as the size of a data set increases, the uncertainty associated 
with the posterior distribution over model parameters decreases. Make use of the 
matrix identity (Appendix C) 


(M~'v) (vTM!) 
1+v'M-!v 


(M+vvT) =M! (3.110) 
to show that the uncertainty c% (x) associated with the linear regression function 
given by (3.59) satisfies 


oN (X) < an (x). (3.111) 


(xx) We saw in Section 2.3.6 that the conjugate prior for a Gaussian distribution 
with unknown mean and unknown precision (inverse variance) is a normal-gamma 
distribution. This property also holds for the case of the conditional Gaussian dis- 
tribution p(t|x, w, 3) of the linear regression model. If we consider the likelihood 
function (3.10), then the conjugate prior for w and £ is given by 


p(w, 3) = N(w|mo, 3~1So)Gam(Glao, bo). (3.112) 
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3.13 


3.14 


3.15 


3.16 


3.17 


3.18 


3.19 


Show that the corresponding posterior distribution takes the same functional form, 
so that 
p(w, BIÐ = N(w|my, 6-'Sv)Gam(Blay, bn) (3.113) 


and find expressions for the posterior parameters my, Sy, ay, and by. 


(xx) Show that the predictive distribution p(t|x, t) for the model discussed in Ex- 
ercise 3.12 is given by a Student’s t-distribution of the form 


p(t|x, t) = St(t|y, à, v) (3.114) 
and obtain expressions for u, À and v. 


(xx) In this exercise, we explore in more detail the properties of the equivalent 
kernel defined by (3.62), where Sy is defined by (3.54). Suppose that the basis 
functions $;(x) are linearly independent and that the number N of data points is 
greater than the number M of basis functions. Furthermore, let one of the basis 
functions be constant, say ¢9(x) = 1. By taking suitable linear combinations of 
these basis functions, we can construct a new basis set 7; (x) spanning the same 
space but that are orthonormal, so that 


N 
S > bj (Xn) Pe (Xn) = Ijk (3.115) 


n=1 


where J, is defined to be 1 if j = k and 0 otherwise, and we take wo(x) = 1. Show 
that for a = 0, the equivalent kernel can be written as k(x,x’) = (x)T(x’) 
where % = (41,..., Ym)". Use this result to show that the kernel satisfies the 
summation constraint 


N 
So k(x, Xn) =1. (3.116) 
n=1 


(x) [QTY Consider a linear basis function model for regression in which the pa- 
rameters a and 8 are set using the evidence framework. Show that the function 
E(my) defined by (3.82) satisfies the relation 2E(my) = N. 


(xx) Derive the result (3.86) for the log evidence function p(t|a, 3) of the linear 
regression model by making use of (2.115) to evaluate the integral (3.77) directly. 


(x) Show that the evidence function for the Bayesian linear regression model can 
be written in the form (3.78) in which E(w) is defined by (3.79). 


œ» [QTY By completing the square over w, show that the error function (3.79) 
in Bayesian linear regression can be written in the form (3.80). 


(xx) Show that the integration over w in the Bayesian linear regression model gives 
the result (3.85). Hence show that the log marginal likelihood is given by (3.86). 
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3.21 


3.22 


3.23 
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œ» [QTY Starting from (3.86) verify all of the steps needed to show that maxi- 
mization of the log marginal likelihood function (3.86) with respect to a leads to the 
re-estimation equation (3.92). 


(xx) An alternative way to derive the result (3.92) for the optimal value of a in the 
evidence framework is to make use of the identity 


d d 
—|In|A| =Tr{ A~'—A ). 11 
da 2A r( da ) ee 


Prove this identity by considering the eigenvalue expansion of a real, symmetric 
matrix A, and making use of the standard results for the determinant and trace of 
A expressed in terms of its eigenvalues (Appendix C). Then make use of (3.117) to 
derive (3.92) starting from (3.86). 


(xx) Starting from (3.86) verify all of the steps needed to show that maximiza- 
tion of the log marginal likelihood function (3.86) with respect to 8 leads to the 
re-estimation equation (3.95). 


œ» [QQ Show that the marginal probability of the data, in other words the 
model evidence, for the model described in Exercise 3.12 is given by 


1 be T(an) |Sn|?/? 


3.118 
(On) NP BY Tay) Sol! ee 


p(t) = 


by first marginalizing with respect to w and then with respect to 8. 


(xx) Repeat the previous exercise but now use Bayes’ theorem in the form 


oe p(tlw, 3)p(w, p) (3.119) 


p(w, It) 


and then substitute for the prior and posterior distributions and the likelihood func- 
tion in order to derive the result (3.118). 


In the previous chapter, we explored a class of regression models having particularly 
simple analytical and computational properties. We now discuss an analogous class 
of models for solving classification problems. The goal in classification is to take an 
input vector x and to assign it to one of K discrete classes Cy where k = 1,..., K. 
In the most common scenario, the classes are taken to be disjoint, so that each input is 
assigned to one and only one class. The input space is thereby divided into decision 
regions whose boundaries are called decision boundaries or decision surfaces. In 
this chapter, we consider linear models for classification, by which we mean that the 
decision surfaces are linear functions of the input vector x and hence are defined 
by (D — 1)-dimensional hyperplanes within the D-dimensional input space. Data 
sets whose classes can be separated exactly by linear decision surfaces are said to be 
linearly separable. 

For regression problems, the target variable t was simply the vector of real num- 
bers whose values we wish to predict. In the case of classification, there are various 
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ways of using target values to represent class labels. For probabilistic models, the 
most convenient, in the case of two-class problems, is the binary representation in 
which there is a single target variable t € {0,1} such that t = 1 represents class C1 
and t = 0 represents class C2. We can interpret the value of t as the probability that 
the class is C4, with the values of probability taking only the extreme values of 0 and 
1. For K > 2 classes, it is convenient to use a 1-of- coding scheme in which t is 
a vector of length K such that if the class is C}, then all elements tų of t are zero 
except element tj, which takes the value 1. For instance, if we have K = 5 classes, 
then a pattern from class 2 would be given the target vector 


t = (0,1,0,0,0)". (4.1) 


Again, we can interpret the value of t as the probability that the class is C. For 
nonprobabilistic models, alternative choices of target variable representation will 
sometimes prove convenient. 

In Chapter 1, we identified three distinct approaches to the classification prob- 
lem. The simplest involves constructing a discriminant function that directly assigns 
each vector x to a specific class. A more powerful approach, however, models the 
conditional probability distribution p(C;,|x) in an inference stage, and then subse- 
quently uses this distribution to make optimal decisions. By separating inference 
and decision, we gain numerous benefits, as discussed in Section 1.5.4. There are 
two different approaches to determining the conditional probabilities p(C;,|x). One 
technique is to model them directly, for example by representing them as parametric 
models and then optimizing the parameters using a training set. Alternatively, we 
can adopt a generative approach in which we model the class-conditional densities 
given by p(x|C;,), together with the prior probabilities p(C;,) for the classes, and then 
we compute the required posterior probabilities using Bayes’ theorem 


P(X|Ci)P(Ci) 


P(Cx |x) = p(x) 


(4.2) 
We shall discuss examples of all three approaches in this chapter. 

In the linear regression models considered in Chapter 3, the model prediction 
y(x, w) was given by a linear function of the parameters w. In the simplest case, 
the model is also linear in the input variables and therefore takes the form y(x) = 
wTx + wo, so that y is a real number. For classification problems, however, we wish 
to predict discrete class labels, or more generally posterior probabilities that lie in 
the range (0, 1). To achieve this, we consider a generalization of this model in which 
we transform the linear function of w using a nonlinear function f(-) so that 


y(x) =f (wx + wo) ; (4.3) 


In the machine learning literature f(-) is known as an activation function, whereas 
its inverse is called a link function in the statistics literature. The decision surfaces 
correspond to y(x) = constant, so that w'x + wọ = constant and hence the deci- 
sion surfaces are linear functions of x, even if the function f(-) is nonlinear. For this 
reason, the class of models described by (4.3) are called generalized linear models 
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(McCullagh and Nelder, 1989). Note, however, that in contrast to the models used 
for regression, they are no longer linear in the parameters due to the presence of the 
nonlinear function f(-). This will lead to more complex analytical and computa- 
tional properties than for linear regression models. Nevertheless, these models are 
still relatively simple compared to the more general nonlinear models that will be 
studied in subsequent chapters. 

The algorithms discussed in this chapter will be equally applicable if we first 
make a fixed nonlinear transformation of the input variables using a vector of basis 
functions @(x) as we did for regression models in Chapter 3. We begin by consider- 
ing classification directly in the original input space x, while in Section 4.3 we shall 
find it convenient to switch to a notation involving basis functions for consistency 
with later chapters. 


Discriminant Functions 


A discriminant is a function that takes an input vector x and assigns it to one of K 
classes, denoted Cx. In this chapter, we shall restrict attention to linear discriminants, 
namely those for which the decision surfaces are hyperplanes. To simplify the dis- 
cussion, we consider first the case of two classes and then investigate the extension 
to K > 2 classes. 


4.1.1 Two classes 


The simplest representation of a linear discriminant function is obtained by tak- 
ing a linear function of the input vector so that 


y(x) = wx + wo (4.4) 


where w is called a weight vector, and wọ is a bias (not to be confused with bias in 
the statistical sense). The negative of the bias is sometimes called a threshold. An 
input vector x is assigned to class C; if y(x) > 0 and to class C2 otherwise. The cor- 
responding decision boundary is therefore defined by the relation y(x) = 0, which 
corresponds to a (D — 1)-dimensional hyperplane within the D-dimensional input 
space. Consider two points xq and xp both of which lie on the decision surface. 
Because y(x,) = y(xg) = 0, we have wT (x4 — Xg) = 0 and hence the vector w is 
orthogonal to every vector lying within the decision surface, and so w determines the 
orientation of the decision surface. Similarly, if x is a point on the decision surface, 
then y(x) = 0, and so the normal distance from the origin to the decision surface is 
given by 


a =, (4.5) 


We therefore see that the bias parameter wp determines the location of the decision 

surface. These properties are illustrated for the case of D = 2 in Figure 4.1. 
Furthermore, we note that the value of y(x) gives a signed measure of the per- 

pendicular distance r of the point x from the decision surface. To see this, consider 
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Figure 4.1 Illustration of the geometry of a 
linear discriminant function in two dimensions. 
The decision surface, shown in red, is perpen- 
dicular to w, and its displacement from the 
origin is controlled by the bias parameter wo. 
Also, the signed orthogonal distance of a gen- 
eral point x from the decision surface is given 
by y(x)/||wl- 


an arbitrary point x and let x be its orthogonal projection onto the decision surface, 
so that 
w 
x= X]; +r—. (4.6) 
lwil 
Multiplying both sides of this result by wT and adding wo, and making use of y(x) = 
wx + wo and y(x1) = w'x, + wọ = 0, we have 
pau (4.7) 
wil 
This result is illustrated in Figure 4.1. 
As with the linear regression models in Chapter 3, it is sometimes convenient 
to use a more compact notation in which we introduce an additional dummy ‘input’ 
value xo = 1 and then define w = (wo, w) and X = (Zo, x) so that 


y(x) = Ww' x. (4.8) 


In this case, the decision surfaces are D-dimensional hyperplanes passing through 
the origin of the D + 1-dimensional expanded input space. 


4.1.2 Multiple classes 


Now consider the extension of linear discriminants to K > 2 classes. We might 
be tempted be to build a -class discriminant by combining a number of two-class 
discriminant functions. However, this leads to some serious difficulties (Duda and 
Hart, 1973) as we now show. 

Consider the use of K —1 classifiers each of which solves a two-class problem of 
separating points in a particular class Cx from points not in that class. This is known 
as a one-versus-the-rest classifier. The left-hand example in Figure 4.2 shows an 
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Figure 4.2 Attempting to construct a K class discriminant from a set of two class discriminants leads to am- 
biguous regions, shown in green. On the left is an example involving the use of two discriminants designed to 
distinguish points in class Cx from points not in class C;,. On the right is an example involving three discriminant 
functions each of which is used to separate a pair of classes Cx and C;. 


example involving three classes where this approach leads to regions of input space 
that are ambiguously classified. 

An alternative is to introduce K(K — 1)/2 binary discriminant functions, one 
for every possible pair of classes. This is known as a one-versus-one classifier. Each 
point is then classified according to a majority vote amongst the discriminant func- 
tions. However, this too runs into the problem of ambiguous regions, as illustrated 
in the right-hand diagram of Figure 4.2. 

We can avoid these difficulties by considering a single K-class discriminant 
comprising K linear functions of the form 


YR(X) = WEX + wko (4.9) 


and then assigning a point x to class Cx if yx(x) > y;(x) for all j A k. The decision 
boundary between class Cx and class C; is therefore given by yx(x) = y;(x) and 
hence corresponds to a (D — 1)-dimensional hyperplane defined by 


(wk — Wj)’ x + (wko — wjo) = 0. (4.10) 


This has the same form as the decision boundary for the two-class case discussed in 
Section 4.1.1, and so analogous geometrical properties apply. 

The decision regions of such a discriminant are always singly connected and 
convex. To see this, consider two points x4 and xg both of which lie inside decision 
region Rx, as illustrated in Figure 4.3. Any point X that lies on the line connecting 
xX, and xp can be expressed in the form 


x = Axa + (1 — A)xp (4.11) 
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Figure 4.3 


Illustration of the decision regions for a mul- 
ticlass linear discriminant, with the decision 
boundaries shown in red. If two points xa 
and xg both lie inside the same decision re- 
gion Rx, then any point x that lies on the line 
connecting these two points must also lie in 
Rr, and hence the decision region must be 
singly connected and convex. 


where 0 < À < 1. From the linearity of the discriminant functions, it follows that 
Yk(R) = Ayn (Xa) + (1 — A)yn (xB). (4.12) 


Because both xq and xp lie inside Rg, it follows that yx(xa) > y;(xa), and 
Yk(Xp) > y;(Xp), for all j A k, and hence y,(X) > y,;(X), and so X also lies 
inside Rg. Thus Ry is singly connected and convex. 

Note that for two classes, we can either employ the formalism discussed here, 
based on two discriminant functions y;(x) and y2(x), or else use the simpler but 
equivalent formulation described in Section 4.1.1 based on a single discriminant 
function y(x). 

We now explore three approaches to learning the parameters of linear discrimi- 
nant functions, based on least squares, Fisher’s linear discriminant, and the percep- 
tron algorithm. 


4.1.3 Least squares for classification 


In Chapter 3, we considered models that were linear functions of the parame- 
ters, and we saw that the minimization of a sum-of-squares error function led to a 
simple closed-form solution for the parameter values. It is therefore tempting to see 
if we can apply the same formalism to classification problems. Consider a general 
classification problem with K classes, with a 1-of- binary coding scheme for the 
target vector t. One justification for using least squares in such a context is that it 
approximates the conditional expectation E/t|x] of the target values given the input 
vector. For the binary coding scheme, this conditional expectation is given by the 
vector of posterior class probabilities. Unfortunately, however, these probabilities 
are typically approximated rather poorly, indeed the approximations can have values 
outside the range (0,1), due to the limited flexibility of a linear model as we shall 
see shortly. 

Each class C;, is described by its own linear model so that 


yk(x) = wix + wero (4.13) 


where k = 1,..., K. We can conveniently group these together using vector nota- 
tion so that p 
y(x) = W” (4.14) 


Exercise 4.2 


Section 2.3.7 
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where W is a matrix whose kt column comprises the D + 1-dimensional vector 
Wr = (wko, W;,) and X is the corresponding augmented input vector (1, xT)" with 
a dummy input £o = 1. This representation was discussed in detail in Section 3.1. A 
new input x is then assigned to the class for which the output y = w; x is largest. 


We now determine the parameter matrix W by minimizing a sum-of-squares 
error function, as we did for regression in Chapter 3. Consider a training data set 


{Xn, tn} where n = 1,..., N, and define a matrix T whose n‘" row is the vector tT, 


h 


together with a matrix X whose nt? row is XT. The sum-of-squares error function 


can then be written as 


Ep(W) = sr {ew —T)T(XW — T)} . (4.15) 


Setting the derivative with respect to W to zero, and rearranging, we then obtain the 
solution for W in the form 


W = (XTX)! XTT = XİT (4.16) 


where Xt is the pseudo-inverse of the matrix X, as discussed in Section 3.1.1. We 
then obtain the discriminant function in the form 


p a \T 
y(x) = W'x =TT (x') X. (4.17) 


An interesting property of least-squares solutions with multiple target variables 
is that if every target vector in the training set satisfies some linear constraint 


a't, +b=0 (4.18) 


for some constants a and b, then the model prediction for any value of x will satisfy 
the same constraint so that 
aly(x)+b=0. (4.19) 


Thus if we use a 1-of-K coding scheme for K classes, then the predictions made 
by the model will have the property that the elements of y(x) will sum to 1 for any 
value of x. However, this summation constraint alone is not sufficient to allow the 
model outputs to be interpreted as probabilities because they are not constrained to 
lie within the interval (0, 1). 

The least-squares approach gives an exact closed-form solution for the discrimi- 
nant function parameters. However, even as a discriminant function (where we use it 
to make decisions directly and dispense with any probabilistic interpretation) it suf- 
fers from some severe problems. We have already seen that least-squares solutions 
lack robustness to outliers, and this applies equally to the classification application, 
as illustrated in Figure 4.4. Here we see that the additional data points in the right- 
hand figure produce a significant change in the location of the decision boundary, 
even though these point would be correctly classified by the original decision bound- 
ary in the left-hand figure. The sum-of-squares error function penalizes predictions 
that are ‘too correct’ in that they lie a long way on the correct side of the decision 
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Figure 4.4 The left plot shows data from two classes, denoted by red crosses and blue circles, together with 
the decision boundary found by least squares (magenta curve) and also by the logistic regression model (green 
curve), which is discussed later in Section 4.3.2. The right-hand plot shows the corresponding results obtained 


when extra data points are added at the bottom left of the diagram, showing that least squares is highly sensitive 
to outliers, unlike logistic regression. 


boundary. In Section 7.1.2, we shall consider several alternative error functions for 
classification and we shall see that they do not suffer from this difficulty. 

However, problems with least squares can be more severe than simply lack of 
robustness, as illustrated in Figure 4.5. This shows a synthetic data set drawn from 
three classes in a two-dimensional input space (£1, £2), having the property that lin- 
ear decision boundaries can give excellent separation between the classes. Indeed, 
the technique of logistic regression, described later in this chapter, gives a satisfac- 
tory solution as seen in the right-hand plot. However, the least-squares solution gives 
poor results, with only a small region of the input space assigned to the green class. 

The failure of least squares should not surprise us when we recall that it cor- 
responds to maximum likelihood under the assumption of a Gaussian conditional 
distribution, whereas binary target vectors clearly have a distribution that is far from 
Gaussian. By adopting more appropriate probabilistic models, we shall obtain clas- 
sification techniques with much better properties than least squares. For the moment, 
however, we continue to explore alternative nonprobabilistic methods for setting the 
parameters in the linear classification models. 


4.1.4 Fisher’s linear discriminant 


One way to view a linear classification model is in terms of dimensionality 
reduction. Consider first the case of two classes, and suppose we take the D- 
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Figure 4.5 Example of a synthetic data set comprising three classes, with training data points denoted in red 
(x), green (+), and blue (o). Lines denote the decision boundaries, and the background colours denote the 
respective classes of the decision regions. On the left is the result of using a least-squares discriminant. We see 
that the region of input space assigned to the green class is too small and so most of the points from this class 
are misclassified. On the right is the result of using logistic regressions as described in Section 4.3.2 showing 
correct classification of the training data. 


dimensional input vector x and project it down to one dimension using 
aT 
y=w x. (4.20) 


If we place a threshold on y and classify y > —wp as class C4, and otherwise class 
C2, then we obtain our standard linear classifier discussed in the previous section. 
In general, the projection onto one dimension leads to a considerable loss of infor- 
mation, and classes that are well separated in the original D-dimensional space may 
become strongly overlapping in one dimension. However, by adjusting the com- 
ponents of the weight vector w, we can select a projection that maximizes the class 
separation. To begin with, consider a two-class problem in which there are N; points 
of class Cı and N2 points of class C2, so that the mean vectors of the two classes are 
given by 


m; = x » Xn, ms = x 5 Xn. (4.21) 


neECy n EC 


The simplest measure of the separation of the classes, when projected onto w, is the 
separation of the projected class means. This suggests that we might choose w so as 
to maximize 

Mə — Mı = w' (m2 — mı) (4.22) 


where 
mg = w' my (4.23) 
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Figure 4.6 The left plot shows samples from two classes (depicted in red and blue) along with the histograms 
resulting from projection onto the line joining the class means. Note that there is considerable class overlap in 
the projected space. The right plot shows the corresponding projection based on the Fisher linear discriminant, 
showing the greatly improved class separation. 


Appendix E 
Exercise 4.4 


Exercise 4.5 


is the mean of the projected data from class C. However, this expression can be 
made arbitrarily large simply by increasing the magnitude of w. To solve this 
problem, we could constrain w to have unit length, so that X`, w? = 1. Using 
a Lagrange multiplier to perform the constrained maximization, we then find that 
w x (mə — mı). There is still a problem with this approach, however, as illustrated 
in Figure 4.6. This shows two classes that are well separated in the original two- 
dimensional space (x1, £2) but that have considerable overlap when projected onto 
the line joining their means. This difficulty arises from the strongly nondiagonal 
covariances of the class distributions. The idea proposed by Fisher is to maximize 
a function that will give a large separation between the projected class means while 
also giving a small variance within each class, thereby minimizing the class overlap. 

The projection formula (4.20) transforms the set of labelled data points in x 
into a labelled set in the one-dimensional space y. The within-class variance of the 
transformed data from class Cx is therefore given by 


s} = X (Ym — me)? (4.24) 


nECk 


where yn = w!x,. We can define the total within-class variance for the whole 
data set to be simply s? + s3. The Fisher criterion is defined to be the ratio of the 
between-class variance to the within-class variance and is given by 
(m2 — mı}? 
J = 4.25 

(e s? + s2 oe 
We can make the dependence on w explicit by using (4.20), (4.23), and (4.24) to 
rewrite the Fisher criterion in the form 
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Jy 
mjad (4.26) 


wTSww 
where Sp is the between-class covariance matrix and is given by 
Sp = (m; — mı )(m — mı)” (4.27) 


and Sw is the total within-class covariance matrix, given by 


Sw = - (xn — m1) (xn — my)" + S (xn — m2)(xXn — mo)™. (4.28) 


neECy nEC2 
Differentiating (4.26) with respect to w, we find that J (w) is maximized when 
(w'Sgw)Sww = (w"Sww)Sgw. (4.29) 


From (4.27), we see that Sgw is always in the direction of (m — m1). Furthermore, 
we do not care about the magnitude of w, only its direction, and so we can drop the 
scalar factors (wTSgw) and (wTSww). Multiplying both sides of (4.29) by SẸ 
we then obtain 

w x Sy (m — m). (4.30) 


Note that if the within-class covariance is isotropic, so that Sw is proportional to the 
unit matrix, we find that w is proportional to the difference of the class means, as 
discussed above. 

The result (4.30) is known as Fisher’s linear discriminant, although strictly it 
is not a discriminant but rather a specific choice of direction for projection of the 
data down to one dimension. However, the projected data can subsequently be used 
to construct a discriminant, by choosing a threshold yo so that we classify a new 
point as belonging to C, if y(x) > yo and classify it as belonging to C2 otherwise. 
For example, we can model the class-conditional densities p(y|C;,) using Gaussian 
distributions and then use the techniques of Section 1.2.4 to find the parameters 
of the Gaussian distributions by maximum likelihood. Having found Gaussian ap- 
proximations to the projected classes, the formalism of Section 1.5.1 then gives an 
expression for the optimal threshold. Some justification for the Gaussian assumption 
comes from the central limit theorem by noting that y = wTx is the sum of a set of 
random variables. 


4.1.5 Relation to least squares 


The least-squares approach to the determination of a linear discriminant was 
based on the goal of making the model predictions as close as possible to a set of 
target values. By contrast, the Fisher criterion was derived by requiring maximum 
class separation in the output space. It is interesting to see the relationship between 
these two approaches. In particular, we shall show that, for the two-class problem, 
the Fisher criterion can be obtained as a special case of least squares. 

So far we have considered 1-of- coding for the target values. If, however, we 
adopt a slightly different target coding scheme, then the least-squares solution for 
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Exercise 4.6 


the weights becomes equivalent to the Fisher solution (Duda and Hart, 1973). In 

particular, we shall take the targets for class C4 to be N/Nj, where N; is the number 

of patterns in class Cj, and N is the total number of patterns. This target value 

approximates the reciprocal of the prior probability for class Cı. For class C2, we 

shall take the targets to be —N/N2, where N; is the number of patterns in class C3. 
The sum-of-squares error function can be written 


N 


B= 50 (w?Xn + wo ~ tn). (4.31) 


n=1 


Setting the derivatives of E with respect to wo and w to zero, we obtain respectively 


N 
(w?x, +wo—tn) = 0 (4.32) 
n=1 


N 
5 (w"xn + wo — ta) Xn 


I 
S 


(4.33) 


3 
Il 
un 


From (4.32), and making use of our choice of target coding scheme for the t,,, we 
obtain an expression for the bias in the form 


wo = —w'm (4.34) 


where we have used 


N 
N N 
ty =N: N. =0 4.35 
>, ae (4.35) 


and where m is the mean of the total data set and is given by 
i. 1 


After some straightforward algebra, and again making use of the choice of tn, the 
second equation (4.33) becomes 


(sw a NM S») w= N(m, = mov) (4.37) 


where Sw is defined by (4.28), Sg is defined by (4.27), and we have substituted for 
the bias using (4.34). Using (4.27), we note that Sgw is always in the direction of 
(mə — mı). Thus we can write 


w œx SẸ (m — mı) (4.38) 


where we have ignored irrelevant scale factors. Thus the weight vector coincides 
with that found from the Fisher criterion. In addition, we have also found an expres- 
sion for the bias value wo given by (4.34). This tells us that a new vector x should be 
classified as belonging to class C, if y(x) = w'(x—m) > 0 and class C2 otherwise. 
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4.1.6 Fisher’s discriminant for multiple classes 


We now consider the generalization of the Fisher discriminant to K > 2 classes, 
and we shall assume that the dimensionality D of the input space is greater than the 
number K of classes. Next, we introduce D’ > 1 linear ‘features’ yy = wĮx, where 
k = 1,..., D'. These feature values can conveniently be grouped together to form 
a vector y. Similarly, the weight vectors {w+ } can be considered to be the columns 
of a matrix W, so that 

y = WTx. (4.39) 


Note that again we are not including any bias parameters in the definition of y. The 
generalization of the within-class covariance matrix to the case of K classes follows 
from (4.28) to give 


K 
Sw = 5 S; (4.40) 
k=1 
where 
So = SS (xn — Mk) (Xn —m,)7 (4.41) 
nECk 
1 
m = =) x (4.42) 
Ny nECk 


and N; is the number of patterns in class Cx. In order to find a generalization of the 
between-class covariance matrix, we follow Duda and Hart (1973) and consider first 
the total covariance matrix 


Sp = J a p (4.43) 


where m is the mean of the total data set 


he i< 
m= 5 DX = a7 Me (4.44) 
n=1 k=1 


and N = $, Nj; is the total number of data points. The total covariance matrix can 
be decomposed into the sum of the within-class covariance matrix, given by (4.40) 
and (4.41), plus an additional matrix Sp, which we identify as a measure of the 
between-class covariance 

Sr = Sw + Spg (4.45) 


where 


K 
Ss = 5 > Ng (mx - m) (m — m)”. (4.46) 
k=1 
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These covariance matrices have been defined in the original x-space. We can now 
define similar matrices in the projected D’-dimensional y-space 


K 
sw =) J (¥n— Ma) (¥n — Me)” (4.47) 
k=1 nECk 
and 
K 
ss = X Ne(ur — H)(Hr — H)” (4.48) 
k=1 
where 
1 i 
mST DL Ye B= yD Mhe (4.49) 
nECk k=1 


Again we wish to construct a scalar that is large when the between-class covariance 
is large and when the within-class covariance is small. There are now many possible 
choices of criterion (Fukunaga, 1990). One example is given by 


J(W) = Tr {sọ sB}. (4.50) 


This criterion can then be rewritten as an explicit function of the projection matrix 
W in the form 
J(w) = Tr {(WSwW")"'(WS,W")}. (4.51) 


Maximization of such criteria is straightforward, though somewhat involved, and is 
discussed at length in Fukunaga (1990). The weight values are determined by those 
eigenvectors of Say Sp that correspond to the D’ largest eigenvalues. 

There is one important result that is common to all such criteria, which is worth 
emphasizing. We first note from (4.46) that Sg is composed of the sum of K ma- 
trices, each of which is an outer product of two vectors and therefore of rank 1. In 
addition, only (K — 1) of these matrices are independent as a result of the constraint 
(4.44). Thus, Sp has rank at most equal to (K — 1) and so there are at most (K — 1) 
nonzero eigenvalues. This shows that the projection onto the (K — 1)-dimensional 
subspace spanned by the eigenvectors of Sg does not alter the value of J(w), and 
so we are therefore unable to find more than (K — 1) linear ‘features’ by this means 
(Fukunaga, 1990). 


4.1.7 The perceptron algorithm 


Another example of a linear discriminant model is the perceptron of Rosenblatt 
(1962), which occupies an important place in the history of pattern recognition al- 
gorithms. It corresponds to a two-class model in which the input vector x is first 
transformed using a fixed nonlinear transformation to give a feature vector (x), 
and this is then used to construct a generalized linear model of the form 


y(x) = f (w' (x)) (4.52) 
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where the nonlinear activation function f(-) is given by a step function of the form 


+1, 
=j. 


The vector @(x) will typically include a bias component ¢o(x) = 1. In earlier 
discussions of two-class classification problems, we have focussed on a target coding 
scheme in which t € {0,1}, which is appropriate in the context of probabilistic 
models. For the perceptron, however, it is more convenient to use target values 
t = +1 for class C, and t = —1 for class C2, which matches the choice of activation 
function. 

The algorithm used to determine the parameters w of the perceptron can most 
easily be motivated by error function minimization. A natural choice of error func- 
tion would be the total number of misclassified patterns. However, this does not lead 
to a simple learning algorithm because the error is a piecewise constant function 
of w, with discontinuities wherever a change in w causes the decision boundary to 
move across one of the data points. Methods based on changing w using the gradi- 
ent of the error function cannot then be applied, because the gradient is zero almost 
everywhere. 

We therefore consider an alternative error function known as the perceptron cri- 
terion. To derive this, we note that we are seeking a weight vector w such that 
patterns Xn in class Cı will have w' (xn) > 0, whereas patterns Xn in class C2 
have w' (xn) < 0. Using the t € {—1, +1} target coding scheme it follows that 
we would like all patterns to satisfy w'@(x,,)t, > 0. The perceptron criterion 
associates zero error with any pattern that is correctly classified, whereas for a mis- 
classified pattern x,, it tries to minimize the quantity -w1@(x,,)ty. The perceptron 
criterion is therefore given by 


Ep(w) = — `> W'Ọntn 


nEM 


a>0 
a <0. 


f(a) = (4.53) 


(4.54) 


Frank Rosenblatt 


Seymour Papert. This book was widely misinter- 
preted at the time as showing that neural networks 


1928-1969 


Rosenblatt’s perceptron played an 
important role in the history of ma- 
chine learning. Initially, Rosenblatt 
: simulated the perceptron on an IBM 
oe 704 computer at Cornell in 1957, 
ee errs but by the early 1960s he had built 
special-purpose hardware that provided a direct, par- 
allel implementation of perceptron learning. Many of 
his ideas were encapsulated in “Principles of Neuro- 
dynamics: Perceptrons and the Theory of Brain Mech- 
anisms” published in 1962. Rosenblatt’s work was 
criticized by Marvin Minksy, whose objections were 
published in the book “Perceptrons”, co-authored with 


were fatally flawed and could only learn solutions for 
linearly separable problems. In fact, it only proved 
such limitations in the case of single-layer networks 
such as the perceptron and merely conjectured (in- 
correctly) that they applied to more general network 
models. Unfortunately, however, this book contributed 
to the substantial decline in research funding for neu- 
ral computing, a situation that was not reversed un- 
til the mid-1980s. Today, there are many hundreds, 
if not thousands, of applications of neural networks 
in widespread use, with examples in areas such as 
handwriting recognition and information retrieval be- 
ing used routinely by millions of people. 
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Section 3.1.3 


where M denotes the set of all misclassified patterns. The contribution to the error 
associated with a particular misclassified pattern is a linear function of w in regions 
of w space where the pattern is misclassified and zero in regions where it is correctly 
classified. The total error function is therefore piecewise linear. 

We now apply the stochastic gradient descent algorithm to this error function. 
The change in the weight vector w is then given by 


wit) = w) — nVEp(w) = w +d, tn (4.55) 


where 17 is the learning rate parameter and 7 is an integer that indexes the steps of 
the algorithm. Because the perceptron function y(x, w) is unchanged if we multiply 
w by a constant, we can set the learning rate parameter 7 equal to 1 without of 
generality. Note that, as the weight vector evolves during training, the set of patterns 
that are misclassified will change. 

The perceptron learning algorithm has a simple interpretation, as follows. We 
cycle through the training patterns in turn, and for each pattern x,, we evaluate the 
perceptron function (4.52). If the pattern is correctly classified, then the weight 
vector remains unchanged, whereas if it is incorrectly classified, then for class C4 
we add the vector #(x,,) onto the current estimate of weight vector w while for 
class C2 we subtract the vector @(x,,) from w. The perceptron learning algorithm is 
illustrated in Figure 4.7. 

If we consider the effect of a single update in the perceptron learning algorithm, 
we see that the contribution to the error from a misclassified pattern will be reduced 
because from (4.55) we have 


wt OT bt, = -wT pnta — (Gren) bata < -wT yin (4.56) 


where we have set 7 = 1, and made use of ||¢,,tn||?_ > 0. Of course, this does 
not imply that the contribution to the error function from the other misclassified 
patterns will have been reduced. Furthermore, the change in weight vector may have 
caused some previously correctly classified patterns to become misclassified. Thus 
the perceptron learning rule is not guaranteed to reduce the total error function at 
each stage. 

However, the perceptron convergence theorem states that if there exists an ex- 
act solution (in other words, if the training data set is linearly separable), then the 
perceptron learning algorithm is guaranteed to find an exact solution in a finite num- 
ber of steps. Proofs of this theorem can be found for example in Rosenblatt (1962), 
Block (1962), Nilsson (1965), Minsky and Papert (1969), Hertz et al. (1991), and 
Bishop (1995a). Note, however, that the number of steps required to achieve con- 
vergence could still be substantial, and in practice, until convergence is achieved, 
we will not be able to distinguish between a nonseparable problem and one that is 
simply slow to converge. 

Even when the data set is linearly separable, there may be many solutions, and 
which one is found will depend on the initialization of the parameters and on the or- 
der of presentation of the data points. Furthermore, for data sets that are not linearly 
separable, the perceptron learning algorithm will never converge. 
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Figure 4.7 Illustration of the convergence of the perceptron learning algorithm, showing data points from two 
classes (red and blue) in a two-dimensional feature space (¢1, ¢2). The top left plot shows the initial parameter 
vector w shown as a black arrow together with the corresponding decision boundary (black line), in which the 
arrow points towards the decision region which classified as belonging to the red class. The data point circled 
in green is misclassified and so its feature vector is added to the current weight vector, giving the new decision 
boundary shown in the top right plot. The bottom left plot shows the next misclassified point to be considered, 
indicated by the green circle, and its feature vector is again added to the weight vector giving the decision 
boundary shown in the bottom right plot for which all data points are correctly classified. 
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Figure 4.8 Illustration of the Mark 1 perceptron hardware. The photograph on the left shows how the inputs 
were obtained using a simple camera system in which an input scene, in this case a printed character, was 
illuminated by powerful lights, and an image focussed onto a 20 x 20 array of cadmium sulphide photocells, 
giving a primitive 400 pixel image. The perceptron also had a patch board, shown in the middle photograph, 
which allowed different configurations of input features to be tried. Often these were wired up at random to 
demonstrate the ability of the perceptron to learn without the need for precise wiring, in contrast to a modern 
digital computer. The photograph on the right shows one of the racks of adaptive weights. Each weight was 
implemented using a rotary variable resistor, also called a potentiometer, driven by an electric motor thereby 
allowing the value of the weight to be adjusted automatically by the learning algorithm. 


Aside from difficulties with the learning algorithm, the perceptron does not pro- 
vide probabilistic outputs, nor does it generalize readily to K > 2 classes. The most 
important limitation, however, arises from the fact that (in common with all of the 
models discussed in this chapter and the previous one) it is based on linear com- 
binations of fixed basis functions. More detailed discussions of the limitations of 
perceptrons can be found in Minsky and Papert (1969) and Bishop (1995a). 

Analogue hardware implementations of the perceptron were built by Rosenblatt, 
based on motor-driven variable resistors to implement the adaptive parameters wj. 
These are illustrated in Figure 4.8. The inputs were obtained from a simple camera 
system based on an array of photo-sensors, while the basis functions œ could be 
chosen in a variety of ways, for example based on simple fixed functions of randomly 
chosen subsets of pixels from the input image. Typical applications involved learning 
to discriminate simple shapes or characters. 

At the same time that the perceptron was being developed, a closely related 
system called the adaline, which is short for ‘adaptive linear element’, was being 
explored by Widrow and co-workers. The functional form of the model was the same 
as for the perceptron, but a different approach to training was adopted (Widrow and 
Hoff, 1960; Widrow and Lehr, 1990). 


4.2. Probabilistic Generative Models 


We turn next to a probabilistic view of classification and show how models with 
linear decision boundaries arise from simple assumptions about the distribution of 
the data. In Section 1.5.4, we discussed the distinction between the discriminative 
and the generative approaches to classification. Here we shall adopt a generative 
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Figure 4.9 Plot of the logistic sigmoid function 
a(a) defined by (4.59), shown in 
red, together with the scaled pro- 
bit function (Aa), for A? = 7/8, 
shown in dashed blue, where ®(a) 
is defined by (4.114). The scal- 
ing factor 7/8 is chosen so that the 
derivatives of the two curves are 
equal for a = 0. 


approach in which we model the class-conditional densities p(x|C;,), as well as the 
class priors p(C;,), and then use these to compute posterior probabilities p(C;,|x) 
through Bayes’ theorem. 

Consider first of all the case of two classes. The posterior probability for class 
Cı can be written as 


p(x|Ci)p(C1) 
p(x|C1)p(C1) + p(x|C2)p(C2) 
1 


= eee o(a) (4.57) 


p(Ci|x) 


where we have defined 


_ yp, ROCs )P(C1) 
P(X|C2)p(C2) 
and g(a) is the logistic sigmoid function defined by 
o 1 
~ 1+exp(—a) 


(4.58) 


ola) (4.59) 
which is plotted in Figure 4.9. The term ‘sigmoid’ means S-shaped. This type of 
function is sometimes also called a ‘squashing function’ because it maps the whole 
real axis into a finite interval. The logistic sigmoid has been encountered already 
in earlier chapters and plays an important role in many classification algorithms. It 
satisfies the following symmetry property 


o(—a) = 1- g(a) (4.60) 


as is easily verified. The inverse of the logistic sigmoid is given by 


a=mn(- 7 ) (4.61) 


=i: 


and is known as the logit function. It represents the log of the ratio of probabilities 
In [p(C1|x)/p(C2|x)] for the two classes, also known as the log odds. 
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Note that in (4.57) we have simply rewritten the posterior probabilities in an 
equivalent form, and so the appearance of the logistic sigmoid may seem rather vac- 
uous. However, it will have significance provided a(x) takes a simple functional 
form. We shall shortly consider situations in which a(x) is a linear function of x, in 
which case the posterior probability is governed by a generalized linear model. 

For the case of K > 2 classes, we have 


P(x|Cx)p(Cx) 
>, PCXIC;)p(C;) 
exp(ax) 


2 aa 4.62 
D2 5 exp(a;) Daa 


which is known as the normalized exponential and can be regarded as a multiclass 
generalization of the logistic sigmoid. Here the quantities a; are defined by 


P(Cx|x) 


ak = In p(x|Ck)p(Cr). (4.63) 


The normalized exponential is also known as the softmax function, as it represents 
a smoothed version of the ‘max’ function because, if a, >> a; for all j 4 k, then 
p(Cy|x) = 1, and p(C,|x) ~ 0. 

We now investigate the consequences of choosing specific forms for the class- 
conditional densities, looking first at continuous input variables x and then dis- 
cussing briefly the case of discrete inputs. 


4.2.1 Continuous inputs 


Let us assume that the class-conditional densities are Gaussian and then explore 
the resulting form for the posterior probabilities. To start with, we shall assume that 
all classes share the same covariance matrix. Thus the density for class Cp is given 
by 


P(x|Cx) = rae BILE exp { 5 (x H) Se m)} . (4.64) 
Consider first the case of two classes. From (4.57) and (4.58), we have 
p(C;|\x) = o(w'x + wo) (4.65) 
where we have defined 
w = (m — pm) (4.66) 
wo = -E py J Ton +In on (4.67) 


We see that the quadratic terms in x from the exponents of the Gaussian densities 
have cancelled (due to the assumption of common covariance matrices) leading to 
a linear function of x in the argument of the logistic sigmoid. This result is illus- 
trated for the case of a two-dimensional input space x in Figure 4.10. The resulting 
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Figure 4.10 The left-hand plot shows the class-conditional densities for two classes, denoted red and blue. 
On the right is the corresponding posterior probability p(Ci|x), which is given by a logistic sigmoid of a linear 
function of x. The surface in the right-hand plot is coloured using a proportion of red ink given by p(C:|x) anda 
proportion of blue ink given by p(C2|x) = 1 — p(C1 |x). 


decision boundaries correspond to surfaces along which the posterior probabilities 
p(Ck|x) are constant and so will be given by linear functions of x, and therefore 
the decision boundaries are linear in input space. The prior probabilities p(C;,) enter 
only through the bias parameter wo so that changes in the priors have the effect of 
making parallel shifts of the decision boundary and more generally of the parallel 
contours of constant posterior probability. 

For the general case of K classes we have, from (4.62) and (4.63), 


ay (x) = WE xX + wko (4.68) 

where we have defined 
w = Dp, (4.69) 
wko = E + ln p(Ck). (4.70) 


We see that the ap (x) are again linear functions of x as a consequence of the cancel- 
lation of the quadratic terms due to the shared covariances. The resulting decision 
boundaries, corresponding to the minimum misclassification rate, will occur when 
two of the posterior probabilities (the two largest) are equal, and so will be defined 
by linear functions of x, and so again we have a generalized linear model. 

If we relax the assumption of a shared covariance matrix and allow each class- 
conditional density p(x|C;,) to have its own covariance matrix Xx, then the earlier 
cancellations will no longer occur, and we will obtain quadratic functions of x, giv- 
ing rise to a quadratic discriminant. The linear and quadratic decision boundaries 
are illustrated in Figure 4.11. 
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Figure 4.11 The left-hand plot shows the class-conditional densities for three classes each having a Gaussian 
distribution, coloured red, green, and blue, in which the red and green classes have the same covariance matrix. 
The right-hand plot shows the corresponding posterior probabilities, in which the RGB colour vector represents 
the posterior probabilities for the respective three classes. The decision boundaries are also shown. Notice that 
the boundary between the red and green classes, which have the same covariance matrix, is linear, whereas 
those between the other pairs of classes are quadratic. 


4.2.2 Maximum likelihood solution 


Once we have specified a parametric functional form for the class-conditional 
densities p(x|C;,), we can then determine the values of the parameters, together with 
the prior class probabilities p(C;,), using maximum likelihood. This requires a data 
set comprising observations of x along with their corresponding class labels. 

Consider first the case of two classes, each having a Gaussian class-conditional 
density with a shared covariance matrix, and suppose we have a data set {Xn, tn } 
where n = 1,..., N. Here t,, = 1 denotes class C, and t,, = 0 denotes class Cy. We 
denote the prior class probability p(C1) = 7, so that p(C2) = 1 — r. For a data point 
Xn from class C1, we have t,, = 1 and hence 


P(Xn, C1) = pC, ox, (C1) = aN (xnl H, ©). 


Similarly for class C2, we have t,, = 0 and hence 


P(XnsC2) = p(Co)p(%n|C2) = (1 — 7) (xn| u2, 2). 
Thus the likelihood function is given by 


N 
p(t|7, fy, Ma, X) = II [TN (Xn| Hy, =)" [(1 = m)N (Xn| Mo, zy (4.71) 


n=1 


where t = (t1,...,t N). As usual, it is convenient to maximize the log of the 
likelihood function. Consider first the maximization with respect to 7. The terms in 
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the log likelihood function that depend on 7 are 


N 


S_ {tn lnm + (1 — tn) In(1 — =}. (4.72) 


n=1 


Setting the derivative with respect to 7 equal to zero and rearranging, we obtain 


nr=— Sot M a (4.73) 


where N; denotes the total number of data points in class C4, and Na denotes the total 
number of data points in class C2. Thus the maximum likelihood estimate for 7 is 
simply the fraction of points in class C, as expected. This result is easily generalized 
to the multiclass case where again the maximum likelihood estimate of the prior 
probability associated with class Cx is given by the fraction of the training set points 
Exercise 4.9 assigned to that class. 

Now consider the maximization with respect to u. Again we can pick out of 

the log likelihood function those terms that depend on jz, giving 


N N 
1 
` tn mN (xn|u, 2) = = ` tn(Xn — h) ET (Xn — p1) + const. (4.74) 
n=1 


n=1 


Setting the derivative with respect to jz, to zero and rearranging, we obtain 


1 N 
= — baxa (4.75) 
Hı N, >, 


which is simply the mean of all the input vectors x,, assigned to class Cı. By a 
similar argument, the corresponding result for p, is given by 


1 
by = — Y (l -tn)Xn (4.76) 


which again is the mean of all the input vectors x,, assigned to class C2. 
Finally, consider the maximum likelihood solution for the shared covariance 
matrix X. Picking out the terms in the log likelihood function that depend on ©, we 


have 
ia aes 
X Ty 
T3 ER D tal n — p) = (Xn — My) 
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where we have defined 


N- N. 
s = “s + s, (4.78) 
1 
Ss, = a (xn = Hi ee) (4.79) 
1 
n€C, 
l T 
S> = N (Xn — Ho)(Xn — Ho) . (4.80) 
2 nEC2 


Using the standard result for the maximum likelihood solution for a Gaussian distri- 
bution, we see that © = S, which represents a weighted average of the covariance 
matrices associated with each of the two classes separately. 

This result is easily extended to the K class problem to obtain the corresponding 
maximum likelihood solutions for the parameters in which each class-conditional 
density is Gaussian with a shared covariance matrix. Note that the approach of fitting 
Gaussian distributions to the classes is not robust to outliers, because the maximum 
likelihood estimation of a Gaussian is not robust. 


4.2.3 Discrete features 


Let us now consider the case of discrete feature values x;. For simplicity, we 
begin by looking at binary feature values x; € {0,1} and discuss the extension to 
more general discrete features shortly. If there are D inputs, then a general distribu- 
tion would correspond to a table of 2? numbers for each class, containing 2? — 1 
independent variables (due to the summation constraint). Because this grows expo- 
nentially with the number of features, we might seek a more restricted representa- 
tion. Here we will make the naive Bayes assumption in which the feature values are 
treated as independent, conditioned on the class C. Thus we have class-conditional 
distributions of the form 


p(x|C) = Tv 1 — pri) (4.81) 


which contain D independent parameters for each class. Substituting into (4.63) then 
gives 


= {xiln uki + (1 — xi) In(1 — pei) } + In p(Cp) (4.82) 


which again are linear a of the input values x;. For the case of K = 2 classes, 
we can alternatively consider the logistic sigmoid formulation given by (4.57). Anal- 
ogous results are obtained for discrete variables each of which can take M > 2 
states. 


4.2.4 Exponential family 


As we have seen, for both Gaussian distributed and discrete inputs, the posterior 
class probabilities are given by generalized linear models with logistic sigmoid (K = 


4.3. 
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2 classes) or softmax (K > 2 classes) activation functions. These are particular cases 
of a more general result obtained by assuming that the class-conditional densities 
p(x|Ck) are members of the exponential family of distributions. 

Using the form (2.194) for members of the exponential family, we see that the 
distribution of x can be written in the form 


p(x|Ax) = h(x)g(Ax) exp { A; u(x) } ; (4.83) 


We now restrict attention to the subclass of such distributions for which u(x) = x. 
Then we make use of (2.236) to introduce a scaling parameter s, so that we obtain 
the restricted set of exponential family class-conditional densities of the form 


1 1 1 
p(x|Ax, s) = zh (2x) g(Ax) exp {abet (4.84) 


Note that we are allowing each class to have its own parameter vector A;, but we are 
assuming that the classes share the same scale parameter s. 

For the two-class problem, we substitute this expression for the class-conditional 
densities into (4.58) and we see that the posterior class probability is again given by 
a logistic sigmoid acting on a linear function a(x) which is given by 


a(x) = (Ay — à2)"x + Ing(A1) — Ing(Az) + In p(C1) — In p(C2). (4.85) 


Similarly, for the -class problem, we substitute the class-conditional density ex- 
pression into (4.63) to give 


ap(x) = A} x + Ing(Ax) + Inp(Cy) (4.86) 


and so again is a linear function of x. 


Probabilistic Discriminative Models 


For the two-class classification problem, we have seen that the posterior probability 
of class Cı can be written as a logistic sigmoid acting on a linear function of x, for a 
wide choice of class-conditional distributions p(x|C;,). Similarly, for the multiclass 
case, the posterior probability of class C;, is given by a softmax transformation of a 
linear function of x. For specific choices of the class-conditional densities p(x|C;,), 
we have used maximum likelihood to determine the parameters of the densities as 
well as the class priors p(C;,) and then used Bayes’ theorem to find the posterior class 
probabilities. 

However, an alternative approach is to use the functional form of the generalized 
linear model explicitly and to determine its parameters directly by using maximum 
likelihood. We shall see that there is an efficient algorithm finding such solutions 
known as iterative reweighted least squares, or IRLS. 

The indirect approach to finding the parameters of a generalized linear model, 
by fitting class-conditional densities and class priors separately and then applying 
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Figure 4.12 Illustration of the role of nonlinear basis functions in linear classification models. The left plot 
shows the original input space (x1, x2) together with data points from two classes labelled red and blue. Two 
‘Gaussian’ basis functions ¢1(x) and ¢2(x) are defined in this space with centres shown by the green crosses 
and with contours shown by the green circles. The right-hand plot shows the corresponding feature space 
(¢1, ¢2) together with the linear decision boundary obtained given by a logistic regression model of the form 
discussed in Section 4.3.2. This corresponds to a nonlinear decision boundary in the original input space, 
shown by the black curve in the left-hand plot. 


Bayes’ theorem, represents an example of generative modelling, because we could 
take such a model and generate synthetic data by drawing values of x from the 
marginal distribution p(x). In the direct approach, we are maximizing a likelihood 
function defined through the conditional distribution p(C;|x), which represents a 
form of discriminative training. One advantage of the discriminative approach is 
that there will typically be fewer adaptive parameters to be determined, as we shall 
see shortly. It may also lead to improved predictive performance, particularly when 
the class-conditional density assumptions give a poor approximation to the true dis- 
tributions. 


4.3.1 Fixed basis functions 


So far in this chapter, we have considered classification models that work di- 
rectly with the original input vector x. However, all of the algorithms are equally 
applicable if we first make a fixed nonlinear transformation of the inputs using a 
vector of basis functions @(x). The resulting decision boundaries will be linear in 
the feature space @, and these correspond to nonlinear decision boundaries in the 
original x space, as illustrated in Figure 4.12. Classes that are linearly separable 
in the feature space @(x) need not be linearly separable in the original observation 
space x. Note that as in our discussion of linear models for regression, one of the 
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basis functions is typically set to a constant, say ¢9(x) = 1, so that the correspond- 
ing parameter wo plays the role of a bias. For the remainder of this chapter, we shall 
include a fixed basis function transformation @(x), as this will highlight some useful 
similarities to the regression models discussed in Chapter 3. 

For many problems of practical interest, there is significant overlap between 
the class-conditional densities p(x|C;,). This corresponds to posterior probabilities 
p(C;|x), which, for at least some values of x, are not 0 or 1. In such cases, the opti- 
mal solution is obtained by modelling the posterior probabilities accurately and then 
applying standard decision theory, as discussed in Chapter 1. Note that nonlinear 
transformations @(x) cannot remove such class overlap. Indeed, they can increase 
the level of overlap, or create overlap where none existed in the original observation 
space. However, suitable choices of nonlinearity can make the process of modelling 
the posterior probabilities easier. 

Such fixed basis function models have important limitations, and these will be 
resolved in later chapters by allowing the basis functions themselves to adapt to the 
data. Notwithstanding these limitations, models with fixed nonlinear basis functions 
play an important role in applications, and a discussion of such models will intro- 
duce many of the key concepts needed for an understanding of their more complex 
counterparts. 


4.3.2 Logistic regression 


We begin our treatment of generalized linear models by considering the problem 
of two-class classification. In our discussion of generative approaches in Section 4.2, 
we saw that under rather general assumptions, the posterior probability of class Cı 
can be written as a logistic sigmoid acting on a linear function of the feature vector 
@ so that 


PCIe) = yle) = o (w') (4.87) 


with p(C2|p) = 1 — p(C.|). Here o(-) is the logistic sigmoid function defined by 
(4.59). In the terminology of statistics, this model is known as logistic regression, 
although it should be emphasized that this is a model for classification rather than 
regression. 

For an /-dimensional feature space œ, this model has M adjustable parameters. 
By contrast, if we had fitted Gaussian class conditional densities using maximum 
likelihood, we would have used 2M parameters for the means and M(M + 1)/2 
parameters for the (shared) covariance matrix. Together with the class prior p(C1), 
this gives a total of M (M +5) /2+1 parameters, which grows quadratically with M, 
in contrast to the linear dependence on M of the number of parameters in logistic 
regression. For large values of M, there is a clear advantage in working with the 
logistic regression model directly. 

We now use maximum likelihood to determine the parameters of the logistic 
regression model. To do this, we shall make use of the derivative of the logistic sig- 
moid function, which can conveniently be expressed in terms of the sigmoid function 
itself d 

o 


n~ o(1—o). (4.88) 
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For a data set {,,,t,}, where tn E {0,1} and d, = (Xn), with n = 


nm? 


1,..., N, the likelihood function can be written 
p(t|w) = I yim {1 — ya} (4.89) 
where t = (tı,..., ty)" and yn = p(Ci|¢,,). As usual, we can define an error 


function by taking the negative logarithm of the likelihood, which gives the cross- 
entropy error function in the form 


E(w) = —Inp(t|w) = -5 {tr In yn + (1 — tn) ln(1 — yn)} (4.90) 


where yn = o(a,) and a, = w'@,,. Taking the gradient of the error function with 
respect to w, we obtain 


N 


VE(w) =X (un — tn) bn (4.91) 


n=1 


where we have made use of (4.88). We see that the factor involving the derivative 
of the logistic sigmoid has cancelled, leading to a simplified form for the gradient 
of the log likelihood. In particular, the contribution to the gradient from data point 
n is given by the ‘error’ yn — tn between the target value and the prediction of the 
model, times the basis function vector @,,. Furthermore, comparison with (3.13) 
shows that this takes precisely the same form as the gradient of the sum-of-squares 
error function for the linear regression model. 

If desired, we could make use of the result (4.91) to give a sequential algorithm 
in which patterns are presented one at a time, in which each of the weight vectors is 
updated using (3.22) in which VE, is the n*™ term in (4.91). 

It is worth noting that maximum likelihood can exhibit severe over-fitting for 
data sets that are linearly separable. This arises because the maximum likelihood so- 
lution occurs when the hyperplane corresponding to o = 0.5, equivalent to wT = 
0, separates the two classes and the magnitude of w goes to infinity. In this case, the 
logistic sigmoid function becomes infinitely steep in feature space, corresponding to 
a Heaviside step function, so that every training point from each class k is assigned 
a posterior probability p(C;,|x) = 1. Furthermore, there is typically a continuum 
of such solutions because any separating hyperplane will give rise to the same pos- 
terior probabilities at the training data points, as will be seen later in Figure 10.13. 
Maximum likelihood provides no way to favour one such solution over another, and 
which solution is found in practice will depend on the choice of optimization algo- 
rithm and on the parameter initialization. Note that the problem will arise even if 
the number of data points is large compared with the number of parameters in the 
model, so long as the training data set is linearly separable. The singularity can be 
avoided by inclusion of a prior and finding a MAP solution for w, or equivalently by 
adding a regularization term to the error function. 
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4.3.3 Iterative reweighted least squares 


In the case of the linear regression models discussed in Chapter 3, the maxi- 
mum likelihood solution, on the assumption of a Gaussian noise model, leads to a 
closed-form solution. This was a consequence of the quadratic dependence of the 
log likelihood function on the parameter vector w. For logistic regression, there 
is no longer a closed-form solution, due to the nonlinearity of the logistic sigmoid 
function. However, the departure from a quadratic form is not substantial. To be 
precise, the error function is concave, as we shall see shortly, and hence has a unique 
minimum. Furthermore, the error function can be minimized by an efficient iterative 
technique based on the Newton-Raphson iterative optimization scheme, which uses a 
local quadratic approximation to the log likelihood function. The Newton-Raphson 
update, for minimizing a function E(w), takes the form (Fletcher, 1987; Bishop and 
Nabney, 2008) 

wew) — wold) _ H-1V E(w). (4.92) 
where H is the Hessian matrix whose elements comprise the second derivatives of 
E(w) with respect to the components of w. 

Let us first of all apply the Newton-Raphson method to the linear regression 
model (3.3) with the sum-of-squares error function (3.12). The gradient and Hessian 
of this error function are given by 


N 
VE(w) = X (wt, —tn)o, =P Sw- Pt (4.93) 
n=1 
N 
H=VVE(w) = X e, =8'® (4.94) 


th 


where ® is the N x M design matrix, whose n*™ row is given by œ- . The Newton- 


Raphson update then takes the form 
w (new) = wold) = (Tp)! {oT pwd = at) 
= (Tp) tp"t (4.95) 
which we recognize as the standard least-squares solution. Note that the error func- 
tion in this case is quadratic and hence the Newton-Raphson formula gives the exact 
solution in one step. 
Now let us apply the Newton-Raphson update to the cross-entropy error function 


(4.90) for the logistic regression model. From (4.91) we see that the gradient and 
Hessian of this error function are given by 


N 
VE(w) = SO(n —tn)bn = "(y —t) (4.96) 


n=l 


H = VVE(w Da —yn)o,¢, =PTRƏ (4.97) 
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where we have made use of (4.88). Also, we have introduced the N x N diagonal 
matrix R with elements 


We see that the Hessian is no longer constant but depends on w through the weight- 
ing matrix R, corresponding to the fact that the error function is no longer quadratic. 
Using the property 0 < yn < 1, which follows from the form of the logistic sigmoid 
function, we see that u' Hu > 0 for an arbitrary vector u, and so the Hessian matrix 
H is positive definite. It follows that the error function is a concave function of w 
and hence has a unique minimum. 

The Newton-Raphson update formula for the logistic regression model then be- 
comes 


wew) = weld) _ (TR) ST (y = t) 
= (TRE) {PTRƏw "d — ey —t)} 
= (®'R&) STRZ (4.99) 


where Z is an V-dimensional vector with elements 
z= ow!) — Rly — t). (4.100) 


We see that the update formula (4.99) takes the form of a set of normal equations for a 
weighted least-squares problem. Because the weighing matrix R is not constant but 
depends on the parameter vector w, we must apply the normal equations iteratively, 
each time using the new weight vector w to compute a revised weighing matrix 
R. For this reason, the algorithm is known as iterative reweighted least squares, or 
IRLS (Rubin, 1983). As in the weighted least-squares problem, the elements of the 
diagonal weighting matrix R can be interpreted as variances because the mean and 
variance of t in the logistic regression model are given by 


Ei] = o(x)=y (4.101) 
var[t] = E[t?] — Eff]? = o(x) — o(x)? = y(1—y) (4.102) 


where we have used the property t? = t for t € {0,1}. In fact, we can interpret IRLS 
as the solution to a linearized problem in the space of the variable a = wọ. The 
quantity zn, which corresponds to the n‘® element of z, can then be given a simple 
interpretation as an effective target value in this space obtained by making a local 
linear approximation to the logistic sigmoid function around the current operating 
point wl) 


da 
an(w) = an(wel))+ "1 (tn — yn) 
dyn |wi) 
m tn 
= lwd _ (Yn = tn) _ Zn. (4.103) 


Section 4.2 
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4.3.4 Multiclass logistic regression 


In our discussion of generative models for multiclass classification, we have 
seen that for a large class of distributions, the posterior probabilities are given by a 
softmax transformation of linear functions of the feature variables, so that 


exp(ax) 


P(Celb) = ye(P) = (4.104) 
>; exp(aj) 
where the ‘activations’ ap are given by 
ak = w,@. (4.105) 


There we used maximum likelihood to determine separately the class-conditional 
densities and the class priors and then found the corresponding posterior probabilities 
using Bayes’ theorem, thereby implicitly determining the parameters {w;,}. Here we 
consider the use of maximum likelihood to determine the parameters {w;,} of this 
model directly. To do this, we will require the derivatives of yx with respect to all of 
the activations aj. These are given by 


a = Yk(Ikj = yj) (4.106) 


where J;,; are the elements of the identity matrix. 

Next we write down the likelihood function. This is most easily done using 
the 1-of-K coding scheme in which the target vector t,, for a feature vector @,, 
belonging to class Cx is a binary vector with all elements zero except for element k, 
which equals one. The likelihood function is then given by 


p(T\|wi,...,w =i Tl (Clo. = =i Wee bi (4.107) 


n=1k=1 n=1k=1 


where Ynk = Ye(@,,), and T is an N x K matrix of target variables with elements 
tnk. Taking the negative logarithm then gives 


N K 
E(wi,...,wK) = —Inp(T|wi,...,w == nk IN Ynk (4.108) 
nelesi 


which is known as the cross-entropy error function for the multiclass classification 
problem. 

We now take the gradient of the error function with respect to one of the param- 
eter vectors wj. Making use of the result (4.106) for the derivatives of the softmax 
function, we obtain 


N 
Vw; Elw,- WE) = X (Unj — tni) bn (4.109) 


n=1 
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where we have made use of `, tny = 1. Once again, we see the same form arising 
for the gradient as was found for the sum-of-squares error function with the linear 
model and the cross-entropy error for the logistic regression model, namely the prod- 
uct of the error (Yn; — tnj) times the basis function ¢,,. Again, we could use this 
to formulate a sequential algorithm in which patterns are presented one at a time, in 
which each of the weight vectors is updated using (3.22). 

We have seen that the derivative of the log likelihood function for a linear regres- 
sion model with respect to the parameter vector w for a data point n took the form 
of the ‘error’ yn — tn times the feature vector œ. Similarly, for the combination 
of logistic sigmoid activation function and cross-entropy error function (4.90), and 
for the softmax activation function with the multiclass cross-entropy error function 
(4.108), we again obtain this same simple form. This is an example of a more general 
result, as we shall see in Section 4.3.6. 

To find a batch algorithm, we again appeal to the Newton-Raphson update to 
obtain the corresponding IRLS algorithm for the multiclass problem. This requires 
evaluation of the Hessian matrix that comprises blocks of size M x M in which 
block j, k is given by 


N 
Vow Vrs E(W1,- WK) = — D> Yn (Teg — Ung) Pn Pr (4.110) 
n=1 


As with the two-class problem, the Hessian matrix for the multiclass logistic regres- 
sion model is positive definite and so the error function again has a unique minimum. 
Practical details of IRLS for the multiclass case can be found in Bishop and Nabney 
(2008). 


4.3.5 Probit regression 


We have seen that, for a broad range of class-conditional distributions, described 
by the exponential family, the resulting posterior class probabilities are given by a 
logistic (or softmax) transformation acting on a linear function of the feature vari- 
ables. However, not all choices of class-conditional density give rise to such a simple 
form for the posterior probabilities (for instance, if the class-conditional densities are 
modelled using Gaussian mixtures). This suggests that it might be worth exploring 
other types of discriminative probabilistic model. For the purposes of this chapter, 
however, we shall return to the two-class case, and again remain within the frame- 
work of generalized linear models so that 


p(t = lja) = f(a) (4.111) 


where a = w' @, and f(-) is the activation function. 

One way to motivate an alternative choice for the link function is to consider a 
noisy threshold model, as follows. For each input @,,, we evaluate an = w' @,, and 
then we set the target value according to 


a ifan >0 


4.112 
tn =0 otherwise. ( ) 
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Figure 4.13 Schematic example of a probability density p(0) 


shown by the blue curve, given in this example by a mixture 

of two Gaussians, along with its cumulative distribution function 
f(a), shown by the red curve. Note that the value of the blue 0.8 
curve at any point, such as that indicated by the vertical green 
line, corresponds to the slope of the red curve at the same point. 9 ¢ 
Conversely, the value of the red curve at this point corresponds 

to the area under the blue curve indicated by the shaded green 
region. In the stochastic threshold model, the class label takes 9.4 
the value t = 1 if the value of a = wẹ exceeds a threshold, oth- 
erwise it takes the value t = 0. This is equivalent to an activation 9.2 
function given by the cumulative distribution function f(a). 


Exercise 4.21 


If the value of 6 is drawn from a probability density p(@), then the corresponding 
activation function will be given by the cumulative distribution function 


f(a) = a p(0) dé (4.113) 


as illustrated in Figure 4.13. 

As a specific example, suppose that the density p(@) is given by a zero mean, 
unit variance Gaussian. The corresponding cumulative distribution function is given 
by 


®(a) =) N (60, 1) a0 (4.114) 


which is known as the probit function. It has a sigmoidal shape and is compared 
with the logistic sigmoid function in Figure 4.9. Note that the use of a more gen- 
eral Gaussian distribution does not change the model because this is equivalent to 
a re-scaling of the linear coefficients w. Many numerical packages provide for the 
evaluation of a closely related function defined by 


erf(a) = = | exp(—6?/2) d0 (4.115) 
0 


and known as the erf function or error function (not to be confused with the error 
function of a machine learning model). It is related to the probit function by 


(a) = ; fı + zmo) ; (4.116) 


The generalized linear model based on a probit activation function is known as probit 
regression. 

We can determine the parameters of this model using maximum likelihood, by a 
straightforward extension of the ideas discussed earlier. In practice, the results found 
using probit regression tend to be similar to those of logistic regression. We shall, 
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however, find another use for the probit model when we discuss Bayesian treatments 
of logistic regression in Section 4.5. 

One issue that can occur in practical applications is that of outliers, which can 
arise for instance through errors in measuring the input vector x or through misla- 
belling of the target value t. Because such points can lie a long way to the wrong side 
of the ideal decision boundary, they can seriously distort the classifier. Note that the 
logistic and probit regression models behave differently in this respect because the 
tails of the logistic sigmoid decay asymptotically like exp(—) for x — oo, whereas 
for the probit activation function they decay like exp(—z7), and so the probit model 
can be significantly more sensitive to outliers. 

However, both the logistic and the probit models assume the data is correctly 
labelled. The effect of mislabelling is easily incorporated into a probabilistic model 
by introducing a probability € that the target value t has been flipped to the wrong 
value (Opper and Winther, 2000a), leading to a target value distribution for data point 
x of the form 


(1 — €)o(x) + €(1 — o(x)) 
e+ (1 — 2e)o(x) (4.117) 


p(t|x) 


where o(x) is the activation function with input vector x. Here e may be set in 
advance, or it may be treated as a hyperparameter whose value is inferred from the 
data. 


4.3.6 Canonical link functions 


For the linear regression model with a Gaussian noise distribution, the error 
function, corresponding to the negative log likelihood, is given by (3.12). If we take 
the derivative with respect to the parameter vector w of the contribution to the error 
function from a data point n, this takes the form of the ‘error’ yn — tn times the 
feature vector @,,, where yn = w' @,,. Similarly, for the combination of the logistic 
sigmoid activation function and the cross-entropy error function (4.90), and for the 
softmax activation function with the multiclass cross-entropy error function (4.108), 
we again obtain this same simple form. We now show that this is a general result 
of assuming a conditional distribution for the target variable from the exponential 
family, along with a corresponding choice for the activation function known as the 
canonical link function. 

We again make use of the restricted form (4.84) of exponential family distribu- 
tions. Note that here we are applying the assumption of exponential family distribu- 
tion to the target variable t, in contrast to Section 4.2.4 where we applied it to the 
input vector x. We therefore consider conditional distributions of the target variable 
of the form 


p(t|n, s) = h (<) gln) epf T} i (4.118) 


Using the same line of argument as led to the derivation of the result (2.226), we see 
that the conditional mean of t, which we denote by y, is given by 


y = Eft 


d 
n| = E In g(n). (4.119) 


4.4. 
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Thus y and 7 must related, and we denote this relation through 7 = w(y). 

Following Nelder and Wedderburn (1972), we define a generalized linear model 
to be one for which y is a nonlinear function of a linear combination of the input (or 
feature) variables so that 


= f(w"¢) (4.120) 


where f(-) is known as the activation function in the machine learning literature, and 
f—‘(-) is known as the link function in statistics. 

Now consider the log likelihood function for this model, which, as a function of 
n, is given by 


ntn 
In p(t|n, s) = Sonn (talin, s) D {In.g(1m) en s \ + const (4.121) 


n=1 


where we are assuming that all observations share a common scale parameter (which 
corresponds to the noise variance for a Gaussian distribution for instance) and so s 
is independent of n. The derivative of the log likelihood with respect to the model 
parameters w is then given by 


. d dnn dyn 
Vw ln p(t|n, s) = bD ay, 9m) 4 tn ik Ta 
n=1 i n n 
4 
= > S {in — Yn} Y (un) f’ (an) bn (4.122) 


where an = wT@,„, and we have used yn = f (an) together with the result (4.119) 
for Eft|n]. We now see that there is a considerable simplification if we choose a 
particular form for the link function f~'(y) given by 


Hy) = vy) (4.123) 
which gives f(¢(y)) = y and hence f’ wy (y) = 1. Also, because a = f~'(y), 
we have a = y and hence f’(a)i'(y) = 1. In this case, the gradient of the error 
function reduces to 

Vin E(w -iY in ~ tn} On. (4.124) 


For the Gaussian s = 3~', whereas for the logistic model s = 1. 


The Laplace Approximation 


In Section 4.5 we shall discuss the Bayesian treatment of logistic regression. As 
we Shall see, this is more complex than the Bayesian treatment of linear regression 
models, discussed in Sections 3.3 and 3.5. In particular, we cannot integrate exactly 
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over the parameter vector w since the posterior distribution is no longer Gaussian. 
It is therefore necessary to introduce some form of approximation. Later in the 
book we shall consider a range of techniques based on analytical approximations 
and numerical sampling. 

Here we introduce a simple, but widely used, framework called the Laplace ap- 
proximation, that aims to find a Gaussian approximation to a probability density 
defined over a set of continuous variables. Consider first the case of a single contin- 
uous variable z, and suppose the distribution p(z) is defined by 


p(z) = 5 f(z) (4.125) 


where Z = f f(z) dz is the normalization coefficient. We shall suppose that the 
value of Z is unknown. In the Laplace method the goal is to find a Gaussian approx- 
imation q(z) which is centred on a mode of the distribution p(z). The first step is to 
find a mode of p(z), in other words a point zo such that p’(zo) = 0, or equivalently 


=0. (4.126) 


Z=Z0 


A Gaussian distribution has the property that its logarithm is a quadratic function 
of the variables. We therefore consider a Taylor expansion of In f(z) centred on the 
mode zo so that 


In f(z) = In f(z) — 5A(z — 2)? (4.127) 
where 
d2 
A=- i In f(z) = (4.128) 


Note that the first-order term in the Taylor expansion does not appear since Zo is a 
local maximum of the distribution. Taking the exponential we obtain 


f(z) = f (zo) exp {-$6 — a} : (4.129) 


We can then obtain a normalized distribution g(z) by making use of the standard 
result for the normalization of a Gaussian, so that 


1/2 
q(z) = (5) exp {-$6 — a} ; (4.130) 


The Laplace approximation is illustrated in Figure 4.14. Note that the Gaussian 
approximation will only be well defined if its precision A > 0, in other words the 
stationary point zọ must be a local maximum, so that the second derivative of f(z) 
at the point zo is negative. 
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=2 -1 0 1 2 3 4 
Figure 4.14 Illustration of the Laplace approximation applied to the distribution p(z) « exp(—z?/2)o(20z + 4) 
where o(z) is the logistic sigmoid function defined by o(z) = (1 + e~*)~'. The left plot shows the normalized 


distribution p(z) in yellow, together with the Laplace approximation centred on the mode zo of p(z) in red. The 
right plot shows the negative logarithms of the corresponding curves. 


We can extend the Laplace method to approximate a distribution p(z) = f(z)/Z 
defined over an M-dimensional space z. At a stationary point Zo the gradient V f(z) 
will vanish. Expanding around this stationary point we have 


In f(z) ~ In f (zo) — (2 — zo)" A(z — Zo) (4.131) 


where the M x M Hessian matrix A is defined by 


A = — VV h f(z)| (4.132) 


Z=—Zo 


and V is the gradient operator. Taking the exponential of both sides we obtain 


f(z) = f (Zo) exp {302 — zo)" A(z — zo)} ‘ (4.133) 


The distribution q(z) is proportional to f(z) and the appropriate normalization coef- 
ficient can be found by inspection, using the standard result (2.43) for a normalized 
multivariate Gaussian, giving 


[Alt 
q(z) = (Qn)MP2 


exp { 5 (2 Zo) A(z — zo)} =N(z2\z,A~') (4.134) 


where |A| denotes the determinant of A. This Gaussian distribution will be well 
defined provided its precision matrix, given by A, is positive definite, which implies 
that the stationary point z) must be a local maximum, not a minimum or a saddle 
point. 

In order to apply the Laplace approximation we first need to find the mode Zo, 
and then evaluate the Hessian matrix at that mode. In practice a mode will typi- 
cally be found by running some form of numerical optimization algorithm (Bishop 
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and Nabney, 2008). Many of the distributions encountered in practice will be mul- 
timodal and so there will be different Laplace approximations according to which 
mode is being considered. Note that the normalization constant Z of the true distri- 
bution does not need to be known in order to apply the Laplace method. As a result 
of the central limit theorem, the posterior distribution for a model is expected to 
become increasingly better approximated by a Gaussian as the number of observed 
data points is increased, and so we would expect the Laplace approximation to be 
most useful in situations where the number of data points is relatively large. 

One major weakness of the Laplace approximation is that, since it is based on a 
Gaussian distribution, it is only directly applicable to real variables. In other cases 
it may be possible to apply the Laplace approximation to a transformation of the 
variable. For instance if 0 < 7 < oo then we can consider a Laplace approximation 
of Inv. The most serious limitation of the Laplace framework, however, is that 
it is based purely on the aspects of the true distribution at a specific value of the 
variable, and so can fail to capture important global properties. In Chapter 10 we 
shall consider alternative approaches which adopt a more global perspective. 


4.4.1 Model comparison and BIC 


As well as approximating the distribution p(z) we can also obtain an approxi- 
mation to the normalization constant Z. Using the approximation (4.133) we have 


/ f(z) dz 


f (Zo) [ox {-3@ — Zo) A(z — zo)} dz 


7 (27r) M/2 
= fzo) TA 


where we have noted that the integrand is Gaussian and made use of the standard 
result (2.43) for a normalized Gaussian distribution. We can use the result (4.135) to 
obtain an approximation to the model evidence which, as discussed in Section 3.4, 
plays a central role in Bayesian model comparison. 

Consider a data set D and a set of models {M;} having parameters {0;}. For 
each model we define a likelihood function p(D|0;,.M,;). If we introduce a prior 
p(9;|M,) over the parameters, then we are interested in computing the model evi- 
dence p(D|M;) for the various models. From now on we omit the conditioning on 
Mi to keep the notation uncluttered. From Bayes’ theorem the model evidence is 
given by 


Z 


l2 


(4.135) 


p(D) = [ mien) dé. (4.136) 


Identifying f(@) = p(D|@)p(@) and Z = p(D), and applying the result (4.135), we 
obtain 


M 1 
Inp(D) ~ Inp(D|O@map) + In p(O@map) + z (27) E In |A| (4.137) 
be 


Occam factor 
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where Omar is the value of @ at the mode of the posterior distribution, and A is the 
Hessian matrix of second derivatives of the negative log posterior 


The first term on the right hand side of (4.137) represents the log likelihood evalu- 
ated using the optimized parameters, while the remaining three terms comprise the 
‘Occam factor’ which penalizes model complexity. 

If we assume that the Gaussian prior distribution over parameters is broad, and 
that the Hessian has full rank, then we can approximate (4.137) very roughly using 


1 
Inp(D) ~ Inp(D|@map) — 5M In N (4.139) 


where N is the number of data points, M is the number of parameters in 0 and 
we have omitted additive constants. This is known as the Bayesian Information 
Criterion (BIC) or the Schwarz criterion (Schwarz, 1978). Note that, compared to 
AIC given by (1.73), this penalizes model complexity more heavily. 

Complexity measures such as AIC and BIC have the virtue of being easy to 
evaluate, but can also give misleading results. In particular, the assumption that the 
Hessian matrix has full rank is often not valid since many of the parameters are not 
‘well-determined’. We can use the result (4.137) to obtain a more accurate estimate 
of the model evidence starting from the Laplace approximation, as we illustrate in 
the context of neural networks in Section 5.7. 


Bayesian Logistic Regression 


We now turn to a Bayesian treatment of logistic regression. Exact Bayesian infer- 
ence for logistic regression is intractable. In particular, evaluation of the posterior 
distribution would require normalization of the product of a prior distribution and a 
likelihood function that itself comprises a product of logistic sigmoid functions, one 
for every data point. Evaluation of the predictive distribution is similarly intractable. 
Here we consider the application of the Laplace approximation to the problem of 
Bayesian logistic regression (Spiegelhalter and Lauritzen, 1990; MacKay, 1992b). 


4.5.1 Laplace approximation 


Recall from Section 4.4 that the Laplace approximation is obtained by finding 
the mode of the posterior distribution and then fitting a Gaussian centred at that 
mode. This requires evaluation of the second derivatives of the log posterior, which 
is equivalent to finding the Hessian matrix. 

Because we seek a Gaussian representation for the posterior distribution, it is 
natural to begin with a Gaussian prior, which we write in the general form 


p(w) = N(w|mo, So) (4.140) 
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where mo and Sọ are fixed hyperparameters. The posterior distribution over w is 
given by 
p(w|t) x p(w)p(t|w) (4.141) 


where t = (t,,...,t,y)*. Taking the log of both sides, and substituting for the prior 
distribution using (4.140), and for the likelihood function using (4.89), we obtain 


1 = 
Inp(w|t) = = W — mo)" S7 (w — mo) 
N 
+Y {tn ln yn + (1— tn) In(1 — yn)} + const (4.142) 
n=1 


where yn = o(w'@,,). To obtain a Gaussian approximation to the posterior dis- 
tribution, we first maximize the posterior distribution to give the MAP (maximum 
posterior) solution wmap, which defines the mean of the Gaussian. The covariance 
is then given by the inverse of the matrix of second derivatives of the negative log 
likelihood, which takes the form 


N 
Sy =—-VVInp(wit) = 857+ So yn(1— yn) bn bn: (4.143) 


n=1 
The Gaussian approximation to the posterior distribution therefore takes the form 
q(w) = N(w|wuap, Sw). (4.144) 


Having obtained a Gaussian approximation to the posterior distribution, there 
remains the task of marginalizing with respect to this distribution in order to make 
predictions. 


4.5.2 Predictive distribution 


The predictive distribution for class C1, given a new feature vector #(x), is 
obtained by marginalizing with respect to the posterior distribution p(w|t), which is 
itself approximated by a Gaussian distribution g(w) so that 


piCa\6.%) = | r(Cild,w)plwlt)dw = | o(wTd)awydw (4148) 
with the corresponding probability for class C2 given by p(C2|@, t) = 1 — p(C,|¢, t). 


To evaluate the predictive distribution, we first note that the function o(w1@) de- 
pends on w only through its projection onto œ. Denoting a = w' @, we have 


a(w'd) = f 6(a—w' ¢)o(a) da (4.146) 


where 6(-) is the Dirac delta function. From this we obtain 


fowo) dw = faoa da (4.147) 
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where 


p(a) = fse — w" )q(w) dw. (4.148) 


We can evaluate p(a) by noting that the delta function imposes a linear constraint 
on w and so forms a marginal distribution from the joint distribution q(w) by inte- 
grating out all directions orthogonal to @. Because q(w) is Gaussian, we know from 
Section 2.3.2 that the marginal distribution will also be Gaussian. We can evaluate 
the mean and covariance of this distribution by taking moments, and interchanging 
the order of integration over a and w, so that 


Ha = Ela] = [vlad = f cooywtoaw = Wap? (4.149) 


where we have used the result (4.144) for the variational posterior distribution g(w). 
Similarly 


o2? = varla] = frote — Efa]? } da 
f a(w) {(w')* - (mig)? } dw = "Sno. (4.150) 


Note that the distribution of a takes the same form as the predictive distribution 
(3.58) for the linear regression model, with the noise variance set to zero. Thus our 
variational approximation to the predictive distribution becomes 


p(C,|t) = faoa da = [oN alta, 02) aa. (4.151) 


This result can also be derived directly by making use of the results for the marginal 
of a Gaussian distribution given in Section 2.3.2. 

The integral over a represents the convolution of a Gaussian with a logistic sig- 
moid, and cannot be evaluated analytically. We can, however, obtain a good approx- 
imation (Spiegelhalter and Lauritzen, 1990; MacKay, 1992b; Barber and Bishop, 
1998a) by making use of the close similarity between the logistic sigmoid function 
o(a) defined by (4.59) and the probit function ®(a) defined by (4.114). In order to 
obtain the best approximation to the logistic function we need to re-scale the hori- 
zontal axis, so that we approximate o(a) by ®(Aa). We can find a suitable value of 
A by requiring that the two functions have the same slope at the origin, which gives 
à? = 1/8. The similarity of the logistic sigmoid and the probit function, for this 
choice of A, is illustrated in Figure 4.9. 

The advantage of using a probit function is that its convolution with a Gaussian 
can be expressed analytically in terms of another probit function. Specifically we 
can show that 


J SOON alu o?)aa = ğ (= foam} (4.152) 
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We now apply the approximation o(a) ~ (Aa) to the probit functions appearing 
on both sides of this equation, leading to the following approximation for the convo- 
lution of a logistic sigmoid with a Gaussian 


foan o°) da ~ ø (K(o7)p) (4.153) 


where we have defined 
klo?) = (1+ ro? /8)71?. (4.154) 


Applying this result to (4.151) we obtain the approximate predictive distribution 
in the form 


P(Ci|, t) = o (K(07) Ha) (4.155) 


where ju, and o? are defined by (4.149) and (4.150), respectively, and «(o2) is de- 
fined by (4.154). 

Note that the decision boundary corresponding to p(C;|@,t) = 0.5 is given by 
[ta = 0, which is the same as the decision boundary obtained by using the MAP 
value for w. Thus if the decision criterion is based on minimizing misclassifica- 
tion rate, with equal prior probabilities, then the marginalization over w has no ef- 
fect. However, for more complex decision criteria it will play an important role. 
Marginalization of the logistic sigmoid model under a Gaussian approximation to 
the posterior distribution will be illustrated in the context of variational inference in 
Figure 10.13. 


(xx) Given a set of data points {Xn }, we can define the convex hull to be the set of 
all points x given by 


— 5 OnXn (4.156) 


where a, > 0 and ae an = 1. Consider a second set of points {yn } together with 
their corresponding convex hull. By definition, the two sets of points will be linearly 
separable if there exists a vector W and a scalar wo such that W?x,, + wo > 0 for all 
Xn, and Tyn + wo < 0 for all y,,. Show that if their convex hulls intersect, the two 
sets of points cannot be linearly separable, and conversely that if they are linearly 
separable, their convex hulls do not intersect. 


œ» [QTY Consider the minimization of a sum-of-squares error function (4.15), 
and suppose that all of the target vectors in the training set satisfy a linear constraint 


a't, +b=0 (4.157) 


where tn corresponds to the nt” row of the matrix T in (4.15). Show that as a 
consequence of this constraint, the elements of the model prediction y(x) given by 
the least-squares solution (4.17) also satisfy this constraint, so that 


a’y(x)+b=0. (4.158) 
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To do so, assume that one of the basis functions ¢9(x) = 1 so that the corresponding 
parameter wo plays the role of a bias. 


(xx) Extend the result of Exercise 4.2 to show that if multiple linear constraints 
are satisfied simultaneously by the target vectors, then the same constraints will also 
be satisfied by the least-squares prediction of a linear model. 


œ FQ Show that maximization of the class separation criterion given by (4.23) 
with respect to w, using a Lagrange multiplier to enforce the constraint w'w = 1, 
leads to the result that w œ (mə — m1). 


(x) By making use of (4.20), (4.23), and (4.24), show that the Fisher criterion (4.25) 
can be written in the form (4.26). 


(x) Using the definitions of the between-class and within-class covariance matrices 
given by (4.27) and (4.28), respectively, together with (4.34) and (4.36) and the 
choice of target values described in Section 4.1.5, show that the expression (4.33) 
that minimizes the sum-of-squares error function can be written in the form (4.37). 


œ [QTY Show that the logistic sigmoid function (4.59) satisfies the property 
o(—a) = 1 — o(a) and that its inverse is given by o~'(y) = In {y/(1 — y)}. 


(x) Using (4.57) and (4.58), derive the result (4.65) for the posterior class probability 
in the two-class generative model with Gaussian densities, and verify the results 
(4.66) and (4.67) for the parameters w and wo. 


(x) www | Consider a generative classification model for K classes defined by 
prior class probabilities p(C;,) = 7, and general class-conditional densities p(ġ|C+) 
where œ is the input feature vector. Suppose we are given a training data set {@,,, tn } 
where n = 1,..., N, and tn is a binary target vector of length K that uses the 1-of- 
K coding scheme, so that it has components tn; = Jj; if pattern n is from class Cx. 
Assuming that the data points are drawn independently from this model, show that 
the maximum-likelihood solution for the prior probabilities is given by 


N (4.159) 


Tk 


where Nx is the number of data points assigned to class Cx. 


(xx) Consider the classification model of Exercise 4.9 and now suppose that the 
class-conditional densities are given by Gaussian distributions with a shared covari- 
ance matrix, so that 


P(PICK) = N (P| og, X). (4.160) 


Show that the maximum likelihood solution for the mean of the Gaussian distribution 
for class Cx is given by 


N 
1 
= ` tn 4.161 
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4.11 


4.12 


4.13 


4.14 


4.15 


4.16 


which represents the mean of those feature vectors assigned to class Cp. Similarly, 
show that the maximum likelihood solution for the shared covariance matrix is given 
by 


LN 
Sa 5 
y Sk (4.162) 
k=1 
where 
1 N 
Si = AD tne On = Ha) (On = Ma)" (4.163) 
n=1 


Thus ¥ is given by a weighted average of the covariances of the data associated with 
each class, in which the weighting coefficients are given by the prior probabilities of 
the classes. 


(xx) Consider a classification problem with K classes for which the feature vector 
@ has M components each of which can take L discrete states. Let the values of the 
components be represented by a 1-of-L binary coding scheme. Further suppose that, 
conditioned on the class C, the M components of @ are independent, so that the 
class-conditional density factorizes with respect to the feature vector components. 
Show that the quantities a, given by (4.63), which appear in the argument to the 
softmax function describing the posterior class probabilities, are linear functions of 
the components of @. Note that this represents an example of the naive Bayes model 
which is discussed in Section 8.2.2. 


(x) FY Verify the relation (4.88) for the derivative of the logistic sigmoid func- 
tion defined by (4.59). 


(x) FY By making use of the result (4.88) for the derivative of the logistic sig- 
moid, show that the derivative of the error function (4.90) for the logistic regression 
model is given by (4.91). 


(x) Show that for a linearly separable data set, the maximum likelihood solution 
for the logistic regression model is obtained by finding a vector w whose decision 
boundary wT @(x) = 0 separates the classes and then taking the magnitude of w to 
infinity. 


(xx) Show that the Hessian matrix H for the logistic regression model, given by 
(4.97), is positive definite. Here R is a diagonal matrix with elements y,,(1 — Yn), 
and y,, is the output of the logistic regression model for input vector x,,. Hence show 
that the error function is a concave function of w and that it has a unique minimum. 


(x) Consider a binary classification problem in which each observation x, is known 
to belong to one of two classes, corresponding to t = 0 and t = 1, and suppose that 
the procedure for collecting training data is imperfect, so that training points are 
sometimes mislabelled. For every data point x,,, instead of having a value t for the 
class label, we have instead a value m, representing the probability that t, = 1. 
Given a probabilistic model p(t = 1|@), write down the log likelihood function 
appropriate to such a data set. 


4.17 


4.18 


4.19 


4.20 


4.21 


4.22 


4.23 


4.24 


4.25 
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œ EY Show that the derivatives of the softmax activation function (4.104), 
where the a, are defined by (4.105), are given by (4.106). 


(x) Using the result (4.91) for the derivatives of the softmax activation function, 
show that the gradients of the cross-entropy error (4.108) are given by (4.109). 


œ~ FY Write down expressions for the gradient of the log likelihood, as well 
as the corresponding Hessian matrix, for the probit regression model defined in Sec- 
tion 4.3.5. These are the quantities that would be required to train such a model using 
IRLS. 


(xx) Show that the Hessian matrix for the multiclass logistic regression problem, 
defined by (4.110), is positive semidefinite. Note that the full Hessian matrix for 
this problem is of size MK x MK, where M is the number of parameters and K 
is the number of classes. To prove the positive semidefinite property, consider the 
product u” Hu where u is an arbitrary vector of length M K, and then apply Jensen’s 
inequality. 


(x) Show that the probit function (4.114) and the erf function (4.115) are related by 
(4.116). 


(x) Using the result (4.135), derive the expression (4.137) for the log model evi- 
dence under the Laplace approximation. 


œ» PQQ [In this exercise, we derive the BIC result (4.139) starting from the 
Laplace approximation to the model evidence given by (4.137). Show that if the 
prior over parameters is Gaussian of the form p(@) = N (8|m, Vo), the log model 
evidence under the Laplace approximation takes the form 


In p(D) ~ Inp(D|O@ap) — 5 (Omar —m)'V5'(@map — m) in [H| + const 
where H is the matrix of second derivatives of the log likelihood In p(D|@) evaluated 
at map. Now assume that the prior is broad so that Vo 1 is small and the second 
term on the right-hand side above can be neglected. Furthermore, consider the case 
of independent, identically distributed data so that H is the sum of terms one for each 
data point. Show that the log model evidence can then be written approximately in 
the form of the BIC expression (4.139). 


(xx) Use the results from Section 2.3.2 to derive the result (4.151) for the marginal- 
ization of the logistic regression model with respect to a Gaussian posterior distribu- 
tion over the parameters w. 


(xx) Suppose we wish to approximate the logistic sigmoid o(a) defined by (4.59) 
by a scaled probit function ®(Aa), where ®(a) is defined by (4.114). Show that if 
A is chosen so that the derivatives of the two functions are equal at a = 0, then 
X? = 7/8, 
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4.26 (xx) In this exercise, we prove the relation (4.152) for the convolution of a probit 
function with a Gaussian distribution. To do this, show that the derivative of the left- 
hand side with respect to u is equal to the derivative of the right-hand side, and then 
integrate both sides with respect to u and then show that the constant of integration 
vanishes. Note that before differentiating the left-hand side, it is convenient first 
to introduce a change of variable given by a = u + oz so that the integral over a 
is replaced by an integral over z. When we differentiate the left-hand side of the 
relation (4.152), we will then obtain a Gaussian integral over z that can be evaluated 
analytically. 


In Chapters 3 and 4 we considered models for regression and classification that com- 
prised linear combinations of fixed basis functions. We saw that such models have 
useful analytical and computational properties but that their practical applicability 
was limited by the curse of dimensionality. In order to apply such models to large- 
scale problems, it is necessary to adapt the basis functions to the data. 

Support vector machines (SVMs), discussed in Chapter 7, address this by first 
defining basis functions that are centred on the training data points and then selecting 
a subset of these during training. One advantage of SVMs is that, although the 
training involves nonlinear optimization, the objective function is convex, and so the 
solution of the optimization problem is relatively straightforward. The number of 
basis functions in the resulting models is generally much smaller than the number of 
training points, although it is often still relatively large and typically increases with 
the size of the training set. The relevance vector machine, discussed in Section 7.2, 
also chooses a subset from a fixed set of basis functions and typically results in much 
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sparser models. Unlike the SVM it also produces probabilistic outputs, although this 
is at the expense of a nonconvex optimization during training. 

An alternative approach is to fix the number of basis functions in advance but 
allow them to be adaptive, in other words to use parametric forms for the basis func- 
tions in which the parameter values are adapted during training. The most successful 
model of this type in the context of pattern recognition is the feed-forward neural 
network, also known as the multilayer perceptron, discussed in this chapter. In fact, 
‘multilayer perceptron’ is really a misnomer, because the model comprises multi- 
ple layers of logistic regression models (with continuous nonlinearities) rather than 
multiple perceptrons (with discontinuous nonlinearities). For many applications, the 
resulting model can be significantly more compact, and hence faster to evaluate, than 
a support vector machine having the same generalization performance. The price to 
be paid for this compactness, as with the relevance vector machine, is that the like- 
lihood function, which forms the basis for network training, is no longer a convex 
function of the model parameters. In practice, however, it is often worth investing 
substantial computational resources during the training phase in order to obtain a 
compact model that is fast at processing new data. 

The term ‘neural network’ has its origins in attempts to find mathematical rep- 
resentations of information processing in biological systems (McCulloch and Pitts, 
1943; Widrow and Hoff, 1960; Rosenblatt, 1962; Rumelhart et al., 1986). Indeed, 
it has been used very broadly to cover a wide range of different models, many of 
which have been the subject of exaggerated claims regarding their biological plau- 
sibility. From the perspective of practical applications of pattern recognition, how- 
ever, biological realism would impose entirely unnecessary constraints. Our focus in 
this chapter is therefore on neural networks as efficient models for statistical pattern 
recognition. In particular, we shall restrict our attention to the specific class of neu- 
ral networks that have proven to be of greatest practical value, namely the multilayer 
perceptron. 

We begin by considering the functional form of the network model, including 
the specific parameterization of the basis functions, and we then discuss the prob- 
lem of determining the network parameters within a maximum likelihood frame- 
work, which involves the solution of a nonlinear optimization problem. This requires 
the evaluation of derivatives of the log likelihood function with respect to the net- 
work parameters, and we shall see how these can be obtained efficiently using the 
technique of error backpropagation. We shall also show how the backpropagation 
framework can be extended to allow other derivatives to be evaluated, such as the 
Jacobian and Hessian matrices. Next we discuss various approaches to regulariza- 
tion of neural network training and the relationships between them. We also consider 
some extensions to the neural network model, and in particular we describe a gen- 
eral framework for modelling conditional probability distributions known as mixture 
density networks. Finally, we discuss the use of Bayesian treatments of neural net- 
works. Additional background on neural network models can be found in Bishop 
(1995a). 


Exercise 5.1 


5.1. 
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Feed-forward Network Functions 


The linear models for regression and classification discussed in Chapters 3 and 4, re- 
spectively, are based on linear combinations of fixed nonlinear basis functions ¢; (x) 
and take the form 


M 
y(x,w) = f | X` wigi) (5.1) 


where f(-) is a nonlinear activation function in the case of classification and is the 
identity in the case of regression. Our goal is to extend this model by making the 
basis functions ¢;(x) depend on parameters and then to allow these parameters to 
be adjusted, along with the coefficients {w; }, during training. There are, of course, 
many ways to construct parametric nonlinear basis functions. Neural networks use 
basis functions that follow the same form as (5.1), so that each basis function is itself 
a nonlinear function of a linear combination of the inputs, where the coefficients in 
the linear combination are adaptive parameters. 

This leads to the basic neural network model, which can be described a series 
of functional transformations. First we construct M linear combinations of the input 
variables x1,..., £p in the form 


n= > wizi nA wy (5.2) 


where j = 1,..., M, and the superscript (1) indicates that the corresponding wine 


eters are in the first ‘layer’ of the network. We shall refer to the parameters wG da 


weights and the parameters w as biases, following the nomenclature of Chapter 3. 


The quantities a; are known as activations. Each of them is then transformed using 
a differentiable, nonlinear activation function h(-) to give 


These quantities correspond to the outputs of the basis functions in (5.1) that, in the 
context of neural networks, are called hidden units. The nonlinear functions h(-) are 
generally chosen to be sigmoidal functions such as the logistic sigmoid or the ‘tanh’ 
function. Following (5.1), these values are again linearly combined to give output 
unit activations 


-5 Wee Zj+ 2 (5.4) 
where k = 1,..., K, and K is the total number of outputs. This transformation cor- 


responds to the second layer of the network, and again the w? are bias parameters. 


Finally, the output unit activations are transformed using an appropriate activation 
function to give a set of network outputs yx. The choice of activation function is 
determined by the nature of the data and the assumed distribution of target variables 
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Figure 5.1 


Network diagram for the two- hidden units 
layer neural network corre- 
sponding to (5.7). The input, 
hidden, and output variables 
are represented by nodes, and TD () 
the weight parameters are rep- 
resented by links between the 

nodes, in which the bias pa- . 
rameters are denoted by links 'Nputs 
coming from additional input 

and hidden variables x) and 

zo. Arrows denote the direc- zı 
tion of information flow through 

the network during forward 
propagation. 


YK 


outputs 


and follows the same considerations as for linear models discussed in Chapters 3 and 
4. Thus for standard regression problems, the activation function is the identity so 
that y = ax. Similarly, for multiple binary classification problems, each output unit 
activation is transformed using a logistic sigmoid function so that 


Yk = olap) (5.5) 
where i 


Finally, for multiclass problems, a softmax activation function of the form (4.62) 
is used. The choice of output unit activation function is discussed in detail in Sec- 
tion 5.2. 

We can combine these various stages to give the overall network function that, 
for sigmoidal output unit activation functions, takes the form 


M D 
nw) =o (3 un (Zutara) g] 6 
j=1 i=1 


where the set of all weight and bias parameters have been grouped together into a 
vector w. Thus the neural network model is simply a nonlinear function from a set 
of input variables {zx;} to a set of output variables {y+ } controlled by a vector w of 
adjustable parameters. 

This function can be represented in the form of a network diagram as shown 
in Figure 5.1. The process of evaluating (5.7) can then be interpreted as a forward 
propagation of information through the network. It should be emphasized that these 
diagrams do not represent probabilistic graphical models of the kind to be consid- 
ered in Chapter 8 because the internal nodes represent deterministic variables rather 
than stochastic ones. For this reason, we have adopted a slightly different graphical 
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notation for the two kinds of model. We shall see later how to give a probabilistic 
interpretation to a neural network. 

As discussed in Section 3.1, the bias parameters in (5.2) can be absorbed into 
the set of weight parameters by defining an additional input variable x) whose value 
is clamped at x9 = 1, so that (5.2) takes the form 


D 
ay = wh ii (5.8) 
i=0 


We can similarly absorb the second-layer biases into the second-layer weights, so 
that the overall network function becomes 


M D 
yk(x,w)=0 X ugh youa . (5.9) 
j=0 i=0 


As can be seen from Figure 5.1, the neural network model comprises two stages 
of processing, each of which resembles the perceptron model of Section 4.1.7, and 
for this reason the neural network is also known as the multilayer perceptron, or 
MLP. A key difference compared to the perceptron, however, is that the neural net- 
work uses continuous sigmoidal nonlinearities in the hidden units, whereas the per- 
ceptron uses step-function nonlinearities. This means that the neural network func- 
tion is differentiable with respect to the network parameters, and this property will 
play a central role in network training. 

If the activation functions of all the hidden units in a network are taken to be 
linear, then for any such network we can always find an equivalent network without 
hidden units. This follows from the fact that the composition of successive linear 
transformations is itself a linear transformation. However, if the number of hidden 
units is smaller than either the number of input or output units, then the transforma- 
tions that the network can generate are not the most general possible linear trans- 
formations from inputs to outputs because information is lost in the dimensionality 
reduction at the hidden units. In Section 12.4.2, we show that networks of linear 
units give rise to principal component analysis. In general, however, there is little 
interest in multilayer networks of linear units. 

The network architecture shown in Figure 5.1 is the most commonly used one 
in practice. However, it is easily generalized, for instance by considering additional 
layers of processing each consisting of a weighted linear combination of the form 
(5.4) followed by an element-wise transformation using a nonlinear activation func- 
tion. Note that there is some confusion in the literature regarding the terminology 
for counting the number of layers in such networks. Thus the network in Figure 5.1 
may be described as a 3-layer network (which counts the number of layers of units, 
and treats the inputs as units) or sometimes as a single-hidden-layer network (which 
counts the number of layers of hidden units). We recommend a terminology in which 
Figure 5.1 is called a two-layer network, because it is the number of layers of adap- 
tive weights that is important for determining the network properties. 

Another generalization of the network architecture is to include skip-layer con- 
nections, each of which is associated with a corresponding adaptive parameter. For 
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Figure 5.2 Example of a neural network having a j 
general feed-forward topology. Note that 22 
each hidden and output unit has an 
associated bias parameter (omitted for £2 yo 
clarity). 
inputs outputs 
Tı yı 
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instance, in a two-layer network these would go directly from inputs to outputs. In 
principle, a network with sigmoidal hidden units can always mimic skip layer con- 
nections (for bounded input values) by using a sufficiently small first-layer weight 
that, over its operating range, the hidden unit is effectively linear, and then com- 
pensating with a large weight value from the hidden unit to the output. In practice, 
however, it may be advantageous to include skip-layer connections explicitly. 

Furthermore, the network can be sparse, with not all possible connections within 
a layer being present. We shall see an example of a sparse network architecture when 
we consider convolutional neural networks in Section 5.5.6. 

Because there is a direct correspondence between a network diagram and its 
mathematical function, we can develop more general network mappings by con- 
sidering more complex network diagrams. However, these must be restricted to a 
feed-forward architecture, in other words to one having no closed directed cycles, to 
ensure that the outputs are deterministic functions of the inputs. This is illustrated 
with a simple example in Figure 5.2. Each (hidden or output) unit in such a network 
computes a function given by 


a= h| Y wyt (5.10) 
J 


where the sum runs over all units that send connections to unit k (and a bias param- 
eter is included in the summation). For a given set of values applied to the inputs of 
the network, successive application of (5.10) allows the activations of all units in the 
network to be evaluated including those of the output units. 

The approximation properties of feed-forward networks have been widely stud- 
ied (Funahashi, 1989; Cybenko, 1989; Hornik et al., 1989; Stinchecombe and White, 
1989; Cotter, 1990; Ito, 1991; Hornik, 1991; Kreinovich, 1991; Ripley, 1996) and 
found to be very general. Neural networks are therefore said to be universal ap- 
proximators. For example, a two-layer network with linear outputs can uniformly 
approximate any continuous function on a compact input domain to arbitrary accu- 
racy provided the network has a sufficiently large number of hidden units. This result 
holds for a wide range of hidden unit activation functions, but excluding polynomi- 
als. Although such theorems are reassuring, the key problem is how to find suitable 
parameter values given a set of training data, and in later sections of this chapter we 


Figure 5.3 Illustration of the ca- 
pability of a multilayer perceptron 
to approximate four different func- 
tions comprising (a) f(x) = x°, (b) 
fæ) = sin(x), (c), f(x) = |æ, 
and (d) f(x) = H(ax) where H(z) 
is the Heaviside step function. In 
each case, N = 50 data points, 
shown as blue dots, have been sam- 
pled uniformly in x over the interval 
(—1,1) and the corresponding val- 
ues of f(x) evaluated. These data 
points are then used to train a two- 
layer network having 3 hidden units 
with ‘tanh’ activation functions and 
linear output units. The resulting 
network functions are shown by the 
red curves, and the outputs of the 
three hidden units are shown by the 
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three dashed curves. 


will show that there exist effective solutions to this problem based on both maximum 
likelihood and Bayesian approaches. 

The capability of a two-layer network to model a broad range of functions is 
illustrated in Figure 5.3. This figure also shows how individual hidden units work 
collaboratively to approximate the final function. The role of hidden units in a simple 
classification problem is illustrated in Figure 5.4 using the synthetic classification 
data set described in Appendix A. 


5.1.1 Weight-space symmetries 


One property of feed-forward networks, which will play a role when we consider 
Bayesian model comparison, is that multiple distinct choices for the weight vector 
w can all give rise to the same mapping function from inputs to outputs (Chen et al., 
1993). Consider a two-layer network of the form shown in Figure 5.1 with M hidden 
units having ‘tanh’ activation functions and full connectivity in both layers. If we 
change the sign of all of the weights and the bias feeding into a particular hidden 
unit, then, for a given input pattern, the sign of the activation of the hidden unit will 
be reversed, because ‘tanh’ is an odd function, so that tanh(—a) = — tanh(a). This 
transformation can be exactly compensated by changing the sign of all of the weights 
leading out of that hidden unit. Thus, by changing the signs of a particular group of 
weights (and a bias), the input-output mapping function represented by the network 
is unchanged, and so we have found two different weight vectors that give rise to 
the same mapping function. For M hidden units, there will be M such ‘sign-flip’ 
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Figure 5.4 Example of the solution of a simple two- 


5.2. 


class classification problem involving 
synthetic data using a neural network 
having two inputs, two hidden units with 
‘tanh’ activation functions, and a single 
output having a logistic sigmoid activa- 
tion function. The dashed blue lines 
show the z = 0.5 contours for each of 
the hidden units, and the red line shows 
the y = 0.5 decision surface for the net- 
work. For comparison, the green line 
denotes the optimal decision boundary 
computed from the distributions used to 
generate the data. 


symmetries, and thus any given weight vector will be one of a set 2™ equivalent 
weight vectors . 

Similarly, imagine that we interchange the values of all of the weights (and the 
bias) leading both into and out of a particular hidden unit with the corresponding 
values of the weights (and bias) associated with a different hidden unit. Again, this 
clearly leaves the network input-output mapping function unchanged, but it corre- 
sponds to a different choice of weight vector. For M hidden units, any given weight 
vector will belong to a set of M! equivalent weight vectors associated with this inter- 
change symmetry, corresponding to the M! different orderings of the hidden units. 
The network will therefore have an overall weight-space symmetry factor of M!2™. 
For networks with more than two layers of weights, the total level of symmetry will 
be given by the product of such factors, one for each layer of hidden units. 

It turns out that these factors account for all of the symmetries in weight space 
(except for possible accidental symmetries due to specific choices for the weight val- 
ues). Furthermore, the existence of these symmetries is not a particular property of 
the ‘tanh’ function but applies to a wide range of activation functions (Kurkova and 
Kainen, 1994). In many cases, these symmetries in weight space are of little practi- 
cal consequence, although in Section 5.7 we shall encounter a situation in which we 
need to take them into account. 


Network Training 


So far, we have viewed neural networks as a general class of parametric nonlinear 
functions from a vector x of input variables to a vector y of output variables. A 
simple approach to the problem of determining the network parameters is to make an 
analogy with the discussion of polynomial curve fitting in Section 1.1, and therefore 
to minimize a sum-of-squares error function. Given a training set comprising a set 
of input vectors {Xn}, where n = 1,..., N, together with a corresponding set of 
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target vectors {t,,}, we minimize the error function 


N 
= 3d ¥(Xa, W) — tn |l. (5.11) 


mio 


However, we can provide a much more general view of network training by first 
giving a probabilistic interpretation to the network outputs. We have already seen 
many advantages of using probabilistic predictions in Section 1.5.4. Here it will also 
provide us with a clearer motivation both for the choice of output unit nonlinearity 
and the choice of error function. 

We start by discussing regression problems, and for the moment we consider 
a single target variable t that can take any real value. Following the discussions 
in Section 1.2.5 and 3.1, we assume that t has a Gaussian distribution with an x- 
dependent mean, which is given by the output of the neural network, so that 


p(t|x, w) =N (tly(x,w), 867") (5.12) 


where (3 is the precision (inverse variance) of the Gaussian noise. Of course this 
is a somewhat restrictive assumption, and in Section 5.6 we shall see how to extend 
this approach to allow for more general conditional distributions. For the conditional 
distribution given by (5.12), it is sufficient to take the output unit activation function 
to be the identity, because such a network can approximate any continuous function 
from x to y. Given a data set of NV independent, identically distributed observations 
X = {x),...,x.}, along with corresponding target values t = {t1,..., ty}, we 
can construct the corresponding likelihood function 


N 
p(t|X, w, 8) = | [ p(tnlxn, w, 9). 
n=1 


Taking the negative logarithm, we obtain the error function 


a N N 
3 dun w) — tn}? 5 InB4 5 In(277) (5.13) 
which can be used to learn the parameters w and 8. In Section 5.7, we shall dis- 
cuss the Bayesian treatment of neural networks, while here we consider a maximum 
likelihood approach. Note that in the neural networks literature, it is usual to con- 
sider the minimization of an error function rather than the maximization of the (log) 
likelihood, and so here we shall follow this convention. Consider first the determi- 
nation of w. Maximizing the likelihood function is equivalent to minimizing the 
sum-of-squares error function given by 


1 N 
=5 Sw wt)? (5.14) 
n=1 
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Exercise 5.2 


Exercise 5.3 


where we have discarded additive and multiplicative constants. The value of w found 
by minimizing E(w) will be denoted wy, because it corresponds to the maximum 
likelihood solution. In practice, the nonlinearity of the network function y(Xn, w) 
causes the error E(w) to be nonconvex, and so in practice local maxima of the 
likelihood may be found, corresponding to local minima of the error function, as 
discussed in Section 5.2.1. 

Having found wym, the value of 8 can be found by minimizing the negative log 
likelihood to give 


ae oe 
2 
— =—) {y(Xn,wmat) = tny. (5.15) 
BML N 2 NI n 

Note that this can be evaluated once the iterative optimization required to find wm, 
is completed. If we have multiple target variables, and we assume that they are inde- 
pendent conditional on x and w with shared noise precision (3, then the conditional 
distribution of the target values is given by 


p(t|x, w) =N (tly(x,w), 67I). (5.16) 


Following the same argument as for a single target variable, we see that the maximum 
likelihood weights are determined by minimizing the sum-of-squares error function 
(5.11). The noise precision is then given by 


1 ees 


But NK >, ly (Xn; Wa) — tnl? (5.17) 
where K is the number of target variables. The assumption of independence can be 
dropped at the expense of a slightly more complex optimization problem. 

Recall from Section 4.3.6 that there is a natural pairing of the error function 
(given by the negative log likelihood) and the output unit activation function. In the 
regression case, we can view the network as having an output activation function that 
is the identity, so that yk = ap. The corresponding sum-of-squares error function 


has the property 
OE 
Dan = Yk 7 tk (5.18) 
ak 
which we shall make use of when discussing error backpropagation in Section 5.3. 
Now consider the case of binary classification in which we have a single target 
variable t such that t = 1 denotes class Cı and t = 0 denotes class C2. Following 
the discussion of canonical link functions in Section 4.3.6, we consider a network 
having a single output whose activation function is a logistic sigmoid 
1 
= = ———_- 5.19 
Yala) 1 + exp(—a) a 
so that 0 < y(x,w) < 1. We can interpret y(x, w) as the conditional probability 
p(Ci|x), with p(C2|x) given by 1 — y(x, w). The conditional distribution of targets 
given inputs is then a Bernoulli distribution of the form 


p(t|x, w) = y(x, w) {1 — y(x, w)} * : (5.20) 
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If we consider a training set of independent observations, then the error function, 
which is given by the negative log likelihood, is then a cross-entropy error function 
of the form 


=-Soth In yn + (1 — ty) In(1 — yn)} (5.21) 


where yn denotes y(x,,, w). Note that there is no analogue of the noise precision 8 
because the target values are assumed to be correctly labelled. However, the model 
is easily extended to allow for labelling errors. Simard et al. (2003) found that using 
the cross-entropy error function instead of the sum-of-squares for a classification 
problem leads to faster training as well as improved generalization. 

If we have K separate binary classifications to perform, then we can use a net- 
work having K outputs each of which has a logistic sigmoid activation function. 
Associated with each output is a binary class label t € {0,1}, where k = 1,..., K. 
If we assume that the class labels are independent, given the input vector, then the 
conditional distribution of the targets is 


p(t|x, w) -JI y(x, w)** [1 — y(x, w) T. (5.22) 


Taking the negative logarithm of the corresponding likelihood function then gives 
the following error function 


N K 
-XX {tnr in yak + (1 — tae) a(l — ynk)} (5.23) 


n=1 k=1 


where yng denotes y,(x,,w). Again, the derivative of the error function with re- 
spect to the activation for a particular output unit takes the form (5.18) just as in the 
regression case. 

It is interesting to contrast the neural network solution to this problem with the 
corresponding approach based on a linear classification model of the kind discussed 
in Chapter 4. Suppose that we are using a standard two-layer network of the kind 
shown in Figure 5.1. We see that the weight parameters in the first layer of the 
network are shared between the various outputs, whereas in the linear model each 
classification problem is solved independently. The first layer of the network can 
be viewed as performing a nonlinear feature extraction, and the sharing of features 
between the different outputs can save on computation and can also lead to improved 
generalization. 

Finally, we consider the standard multiclass classification problem in which each 
input is assigned to one of K mutually exclusive classes. The binary target variables 
tk € {0,1} have a 1-of-k coding scheme indicating the class, and the network 
outputs are interpreted as y,(x,w) = p(t, = 1|x), leading to the following error 


function 
N 


>? ge In yx (Xn, w). (5.24) 
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Figure 5.5 Geometrical view of the error function E(w) as 
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a surface sitting over weight space. Point w4 is 
a local minimum and wz is the global minimum. 
At any point wc, the local gradient of the error 
surface is given by the vector V E. 


W2 VE 


Following the discussion of Section 4.3.4, we see that the output unit activation 
function, which corresponds to the canonical link, is given by the softmax function 


exp(ax(x, w)) 


S| exp(aj(x,w)) 


j 


(5.25) 


Yk(X, w) = 


which satisfies 0 < yọ < 1 and D yk = 1. Note that the y(x, w) are unchanged 
if a constant is added to all of the a(x, w), causing the error function to be constant 
for some directions in weight space. This degeneracy is removed if an appropriate 
regularization term (Section 5.5) is added to the error function. 

Once again, the derivative of the error function with respect to the activation for 
a particular output unit takes the familiar form (5.18). 

In summary, there is a natural choice of both output unit activation function 
and matching error function, according to the type of problem being solved. For re- 
gression we use linear outputs and a sum-of-squares error, for (multiple independent) 
binary classifications we use logistic sigmoid outputs and a cross-entropy error func- 
tion, and for multiclass classification we use softmax outputs with the corresponding 
multiclass cross-entropy error function. For classification problems involving two 
classes, we can use a single logistic sigmoid output, or alternatively we can use a 
network with two outputs having a softmax output activation function. 


5.2.1 Parameter optimization 


We turn next to the task of finding a weight vector w which minimizes the 
chosen function E(w). At this point, it is useful to have a geometrical picture of the 
error function, which we can view as a surface sitting over weight space as shown in 
Figure 5.5. First note that if we make a small step in weight space from w to w+ dw 
then the change in the error function is 5E ~ 6w' V E(w), where the vector V E(w) 
points in the direction of greatest rate of increase of the error function. Because the 
error E(w) is a smooth continuous function of w, its smallest value will occur at a 
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point in weight space such that the gradient of the error function vanishes, so that 
VE(w) =0 (5.26) 


as otherwise we could make a small step in the direction of —V E(w) and thereby 
further reduce the error. Points at which the gradient vanishes are called stationary 
points, and may be further classified into minima, maxima, and saddle points. 

Our goal is to find a vector w such that E(w) takes its smallest value. How- 
ever, the error function typically has a highly nonlinear dependence on the weights 
and bias parameters, and so there will be many points in weight space at which the 
gradient vanishes (or is numerically very small). Indeed, from the discussion in Sec- 
tion 5.1.1 we see that for any point w that is a local minimum, there will be other 
points in weight space that are equivalent minima. For instance, in a two-layer net- 
work of the kind shown in Figure 5.1, with M hidden units, each point in weight 
space is a member of a family of M/!2™ equivalent points. 

Furthermore, there will typically be multiple inequivalent stationary points and 
in particular multiple inequivalent minima. A minimum that corresponds to the 
smallest value of the error function for any weight vector is said to be a global 
minimum. Any other minima corresponding to higher values of the error function 
are said to be local minima. For a successful application of neural networks, it may 
not be necessary to find the global minimum (and in general it will not be known 
whether the global minimum has been found) but it may be necessary to compare 
several local minima in order to find a sufficiently good solution. 

Because there is clearly no hope of finding an analytical solution to the equa- 
tion VE(w) = 0 we resort to iterative numerical procedures. The optimization of 
continuous nonlinear functions is a widely studied problem and there exists an ex- 
tensive literature on how to solve it efficiently. Most techniques involve choosing 
some initial value w°) for the weight vector and then moving through weight space 
in a succession of steps of the form 


wit) = w) 4 Aw (5.27) 


where 7 labels the iteration step. Different algorithms involve different choices for 
the weight vector update Aw‘). Many algorithms make use of gradient information 
and therefore require that, after each update, the value of V E(w) is evaluated at 
the new weight vector w‘+!). In order to understand the importance of gradient 
information, it is useful to consider a local approximation to the error function based 
on a Taylor expansion. 


5.2.2 Local quadratic approximation 


Insight into the optimization problem, and into the various techniques for solv- 
ing it, can be obtained by considering a local quadratic approximation to the error 
function. 

Consider the Taylor expansion of E(w) around some point W in weight space 


E(w) ~ E(w) + (w —Ww)"b4 Lw w) H(w — w) (5.28) 
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where cubic and higher terms have been omitted. Here b is defined to be the gradient 
of E evaluated at w 


b= VE\,.-@ (5.29) 
and the Hessian matrix H = VV E has elements 
OE 
(H); = . (5.30) 


7 Owidw; |= 
From (5.28), the corresponding local approximation to the gradient is given by 
VE~b+H(w-w). (5.31) 


For points w that are sufficiently close to w, these expressions will give reasonable 
approximations for the error and its gradient. 

Consider the particular case of a local quadratic approximation around a point 
w* that is a minimum of the error function. In this case there is no linear term, 
because VE = 0 at w*, and (5.28) becomes 


E(w) = E(w*) + sw — w*)'H(w — w*) (5.32) 


where the Hessian H is evaluated at w*. In order to interpret this geometrically, 
consider the eigenvalue equation for the Hessian matrix 


Hu; = Ayu; (5.33) 
where the eigenvectors u; form a complete orthonormal set (Appendix C) so that 
u;u 


Tuj = dy: (5.34) 


We now expand (w — w*) as a linear combination of the eigenvectors in the form 
wow = So au. (5.35) 
i 


This can be regarded as a transformation of the coordinate system in which the origin 
is translated to the point w*, and the axes are rotated to align with the eigenvectors 
(through the orthogonal matrix whose columns are the u;), and is discussed in more 
detail in Appendix C. Substituting (5.35) into (5.32), and using (5.33) and (5.34), 
allows the error function to be written in the form 


E(w) = E(w*) + ; F Aa. (5.36) 


A matrix H is said to be positive definite if, and only if, 


v'Hv > 0 for all v. (5.37) 
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In the neighbourhood of a min- 
imum w*, the error function 
can be approximated by a 
quadratic. Contours of con- 
stant error are then ellipses 
whose axes are aligned with 
the eigenvectors u; of the Hes- 
sian matrix, with lengths that 
are inversely proportional to the 
square roots of the correspond- 
ing eigenvectors A;. 


Because the eigenvectors {u;} form a complete set, an arbitrary vector v can be 
written in the form 
v=) ni (5.38) 
i 


From (5.33) and (5.34), we then have 


vTHv = 90d (5.39) 


i 


and so H will be positive definite if, and only if, all of its eigenvalues are positive. 
In the new coordinate system, whose basis vectors are given by the eigenvectors 
{u;}, the contours of constant F are ellipses centred on the origin, as illustrated 
in Figure 5.6. For a one-dimensional weight space, a stationary point w* will be a 
minimum if 

OPE 


Ow? 


The corresponding result in D-dimensions is that the Hessian matrix, evaluated at 
w*, should be positive definite. 


> 0. (5.40) 


w* 


5.2.3 Use of gradient information 


As we shall see in Section 5.3, it is possible to evaluate the gradient of an error 
function efficiently by means of the backpropagation procedure. The use of this 
gradient information can lead to significant improvements in the speed with which 
the minima of the error function can be located. We can see why this is so, as follows. 

In the quadratic approximation to the error function, given in (5.28), the error 
surface is specified by the quantities b and H, which contain a total of W(W + 
3) /2 independent elements (because the matrix H is symmetric), where W is the 
dimensionality of w (i.e., the total number of adaptive parameters in the network). 
The location of the minimum of this quadratic approximation therefore depends on 
O(W?) parameters, and we should not expect to be able to locate the minimum until 
we have gathered O(W7) independent pieces of information. If we do not make 
use of gradient information, we would expect to have to perform O(W7) function 
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evaluations, each of which would require O(W) steps. Thus, the computational 
effort needed to find the minimum using such an approach would be O(W*). 

Now compare this with an algorithm that makes use of the gradient information. 
Because each evaluation of VE brings W items of information, we might hope to 
find the minimum of the function in O(W) gradient evaluations. As we shall see, 
by using error backpropagation, each such evaluation takes only O(W) steps and so 
the minimum can now be found in O(W7) steps. For this reason, the use of gradient 
information forms the basis of practical algorithms for training neural networks. 


5.2.4 Gradient descent optimization 


The simplest approach to using gradient information is to choose the weight 
update in (5.27) to comprise a small step in the direction of the negative gradient, so 
that 

wit) = wl — nV E(w”) (5.41) 


where the parameter n > 0 is known as the learning rate. After each such update, the 
gradient is re-evaluated for the new weight vector and the process repeated. Note that 
the error function is defined with respect to a training set, and so each step requires 
that the entire training set be processed in order to evaluate VE. Techniques that 
use the whole data set at once are called batch methods. At each step the weight 
vector is moved in the direction of the greatest rate of decrease of the error function, 
and so this approach is known as gradient descent or steepest descent. Although 
such an approach might intuitively seem reasonable, in fact it turns out to be a poor 
algorithm, for reasons discussed in Bishop and Nabney (2008). 

For batch optimization, there are more efficient methods, such as conjugate gra- 
dients and quasi-Newton methods, which are much more robust and much faster 
than simple gradient descent (Gill et al., 1981; Fletcher, 1987; Nocedal and Wright, 
1999). Unlike gradient descent, these algorithms have the property that the error 
function always decreases at each iteration unless the weight vector has arrived at a 
local or global minimum. 

In order to find a sufficiently good minimum, it may be necessary to run a 
gradient-based algorithm multiple times, each time using a different randomly cho- 
sen starting point, and comparing the resulting performance on an independent vali- 
dation set. 

There is, however, an on-line version of gradient descent that has proved useful 
in practice for training neural networks on large data sets (Le Cun et al., 1989). 
Error functions based on maximum likelihood for a set of independent observations 
comprise a sum of terms, one for each data point 


N 
E(w) = X E,(w). (5.42) 


On-line gradient descent, also known as sequential gradient descent or stochastic 
gradient descent, makes an update to the weight vector based on one data point at a 
time, so that 

= w) — nVEn(w™). (5.43) 


5.3. 
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This update is repeated by cycling through the data either in sequence or by selecting 
points at random with replacement. There are of course intermediate scenarios in 
which the updates are based on batches of data points. 

One advantage of on-line methods compared to batch methods is that the former 
handle redundancy in the data much more efficiently. To see, this consider an ex- 
treme example in which we take a data set and double its size by duplicating every 
data point. Note that this simply multiplies the error function by a factor of 2 and so 
is equivalent to using the original error function. Batch methods will require double 
the computational effort to evaluate the batch error function gradient, whereas on- 
line methods will be unaffected. Another property of on-line gradient descent is the 
possibility of escaping from local minima, since a stationary point with respect to 
the error function for the whole data set will generally not be a stationary point for 
each data point individually. 

Nonlinear optimization algorithms, and their practical application to neural net- 
work training, are discussed in detail in Bishop and Nabney (2008). 


Error Backpropagation 


Our goal in this section is to find an efficient technique for evaluating the gradient 
of an error function E(w) for a feed-forward neural network. We shall see that 
this can be achieved using a local message passing scheme in which information is 
sent alternately forwards and backwards through the network and is known as error 
backpropagation, or sometimes simply as backprop. 

It should be noted that the term backpropagation is used in the neural com- 
puting literature to mean a variety of different things. For instance, the multilayer 
perceptron architecture is sometimes called a backpropagation network. The term 
backpropagation is also used to describe the training of a multilayer perceptron us- 
ing gradient descent applied to a sum-of-squares error function. In order to clarify 
the terminology, it is useful to consider the nature of the training process more care- 
fully. Most training algorithms involve an iterative procedure for minimization of an 
error function, with adjustments to the weights being made in a sequence of steps. At 
each such step, we can distinguish between two distinct stages. In the first stage, the 
derivatives of the error function with respect to the weights must be evaluated. As 
we shall see, the important contribution of the backpropagation technique is in pro- 
viding a computationally efficient method for evaluating such derivatives. Because 
it is at this stage that errors are propagated backwards through the network, we shall 
use the term backpropagation specifically to describe the evaluation of derivatives. 
In the second stage, the derivatives are then used to compute the adjustments to be 
made to the weights. The simplest such technique, and the one originally considered 
by Rumelhart et al. (1986), involves gradient descent. It is important to recognize 
that the two stages are distinct. Thus, the first stage, namely the propagation of er- 
rors backwards through the network in order to evaluate derivatives, can be applied 
to many other kinds of network and not just the multilayer perceptron. It can also be 
applied to error functions other that just the simple sum-of-squares, and to the eval- 
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uation of other derivatives such as the Jacobian and Hessian matrices, as we shall 
see later in this chapter. Similarly, the second stage of weight adjustment using the 
calculated derivatives can be tackled using a variety of optimization schemes, many 
of which are substantially more powerful than simple gradient descent. 


5.3.1 Evaluation of error-function derivatives 


We now derive the backpropagation algorithm for a general network having ar- 
bitrary feed-forward topology, arbitrary differentiable nonlinear activation functions, 
and a broad class of error function. The resulting formulae will then be illustrated 
using a simple layered network structure having a single layer of sigmoidal hidden 
units together with a sum-of-squares error. 

Many error functions of practical interest, for instance those defined by maxi- 
mum likelihood for a set of i.i.d. data, comprise a sum of terms, one for each data 
point in the training set, so that 


E(w) = X En(w). (5.44) 


Here we shall consider the problem of evaluating V En (w) for one such term in the 
error function. This may be used directly for sequential optimization, or the results 
can be accumulated over the training set in the case of batch methods. 

Consider first a simple linear model in which the outputs y;, are linear combina- 
tions of the input variables x; so that 


Yk = >) writ (5.45) 
together with an error function that, for a particular input pattern n, takes the form 
1 2 
En=5 > (Wnt — tnx) (5.46) 


where Ynk = Yk(Xn, W). The gradient of this error function with respect to a weight 
wji is given by 
OE 


Ow ji 


= (Unj tng ns (5.47) 


which can be interpreted as a ‘local’ computation involving the product of an ‘error 
signal’ Ynj — tnj associated with the output end of the link wj; and the variable £n; 
associated with the input end of the link. In Section 4.3.2, we saw how a similar 
formula arises with the logistic sigmoid activation function together with the cross 
entropy error function, and similarly for the softmax activation function together 
with its matching cross-entropy error function. We shall now see how this simple 
result extends to the more complex setting of multilayer feed-forward networks. 

In a general feed-forward network, each unit computes a weighted sum of its 
inputs of the form 


ag= Y n (5.48) 
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where z; is the activation of a unit, or input, that sends a connection to unit j, and wj; 
is the weight associated with that connection. In Section 5.1, we saw that biases can 
be included in this sum by introducing an extra unit, or input, with activation fixed 
at +1. We therefore do not need to deal with biases explicitly. The sum in (5.48) is 
transformed by a nonlinear activation function h(-) to give the activation zj of unit j 
in the form 


Note that one or more of the variables z; in the sum in (5.48) could be an input, and 
similarly, the unit 7 in (5.49) could be an output. 

For each pattern in the training set, we shall suppose that we have supplied the 
corresponding input vector to the network and calculated the activations of all of 
the hidden and output units in the network by successive application of (5.48) and 
(5.49). This process is often called forward propagation because it can be regarded 
as a forward flow of information through the network. 

Now consider the evaluation of the derivative of En» with respect to a weight 
wji. The outputs of the various units will depend on the particular input pattern n. 
However, in order to keep the notation uncluttered, we shall omit the subscript n 
from the network variables. First we note that En depends on the weight wj; only 
via the summed input a; to unit 7. We can therefore apply the chain rule for partial 


derivatives to give 
OE, = OEn Oa; 


; 5.50 
Ow ji Oa; Ow ji ( ) 
We now introduce a useful notation 
OE, 
= Bisi! 
I Oa; ( ) 


where the 6’s are often referred to as errors for reasons we shall see shortly. Using 
(5.48), we can write 
Oa; 
Ow ji 


Substituting (5.51) and (5.52) into (5.50), we then obtain 


= (5.52) 


OEn 


Ow ji 


Equation (5.53) tells us that the required derivative is obtained simply by multiplying 
the value of 6 for the unit at the output end of the weight by the value of z for the unit 
at the input end of the weight (where z = 1 in the case of a bias). Note that this takes 
the same form as for the simple linear model considered at the start of this section. 
Thus, in order to evaluate the derivatives, we need only to calculate the value of ô; 
for each hidden and output unit in the network, and then apply (5.53). 

As we have seen already, for the output units, we have 


Ok = Yk — te (5.54) 
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Figure 5.7 


Illustration of the calculation of ô; for hidden unit j by 
backpropagation of the 6’s from those units k to which 
unit 7 sends connections. The blue arrow denotes the 
direction of information flow during forward propagation, 
and the red arrows indicate the backward propagation 
of error information. 


provided we are using the canonical link as the output-unit activation function. To 
evaluate the 6’s for hidden units, we again make use of the chain rule for partial 
derivatives, 


_ OEn x> OEn Bax 
a Oa; 3 Jay, Oa; oP) 


where the sum runs over all units k to which unit j sends connections. The arrange- 
ment of units and weights is illustrated in Figure 5.7. Note that the units labelled k 
could include other hidden units and/or output units. In writing down (5.55), we are 
making use of the fact that variations in a; give rise to variations in the error func- 
tion only through variations in the variables ap. If we now substitute the definition 
of 6 given by (5.51) into (5.55), and make use of (5.48) and (5.49), we obtain the 
following backpropagation formula 


ô; = h (aj) X wed (5.56) 
k 


which tells us that the value of ô for a particular hidden unit can be obtained by 
propagating the 6’s backwards from units higher up in the network, as illustrated 
in Figure 5.7. Note that the summation in (5.56) is taken over the first index on 
Wrz (corresponding to backward propagation of information through the network), 
whereas in the forward propagation equation (5.10) it is taken over the second index. 
Because we already know the values of the 6’s for the output units, it follows that 
by recursively applying (5.56) we can evaluate the 6’s for all of the hidden units in a 
feed-forward network, regardless of its topology. 
The backpropagation procedure can therefore be summarized as follows. 


Error Backpropagation 


1. Apply an input vector x,, to the network and forward propagate through 
the network using (5.48) and (5.49) to find the activations of all the hidden 
and output units. 


2. Evaluate the ô% for all the output units using (5.54). 


3. Backpropagate the ô’s using (5.56) to obtain 6; for each hidden unit in the 
network. 


4. Use (5.53) to evaluate the required derivatives. 
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For batch methods, the derivative of the total error E can then be obtained by 
repeating the above steps for each pattern in the training set and then summing over 


all patterns: 


OE _ OE, (5.57) 


Ow ji P Ow ji ` 


In the above derivation we have implicitly assumed that each hidden or output unit in 
the network has the same activation function h(-). The derivation is easily general- 
ized, however, to allow different units to have individual activation functions, simply 
by keeping track of which form of h(-) goes with which unit. 


5.3.2 A simple example 


The above derivation of the backpropagation procedure allowed for general 
forms for the error function, the activation functions, and the network topology. In 
order to illustrate the application of this algorithm, we shall consider a particular 
example. This is chosen both for its simplicity and for its practical importance, be- 
cause many applications of neural networks reported in the literature make use of 
this type of network. Specifically, we shall consider a two-layer network of the form 
illustrated in Figure 5.1, together with a sum-of-squares error, in which the output 
units have linear activation functions, so that yg = ax, while the hidden units have 
logistic sigmoid activation functions given by 


h(a) = tanh(a) (5.58) 
where 
et — e 4 
tanh(a) = ————_.. 5.59 
(2) er se t ( ) 


A useful feature of this function is that its derivative can be expressed in a par- 
ticularly simple form: 
h'(a) = 1 — h(a}. (5.60) 
We also consider a standard sum-of-squares error function, so that for pattern n the 
error is given by 


1 K 
En = 5 9 (We te)? (5.61) 


where yx is the activation of output unit k, and tẹ is the corresponding target, for a 
particular input pattern x,,. 

For each pattern in the training set in turn, we first perform a forward propagation 
using 


D 

ap = Sowa (5.62) 
i=0 

zj = tanh(a;) (5.63) 


M 
Yk = Y wa. (5.64) 
j=0 
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Next we compute the 6’s for each output unit using 
Ok = Yk — te. (5.65) 


Then we backpropagate these to obtain ôs for the hidden units using 


K 
55 = (1 — 22) X wrjôr. (5.66) 
k=1 


Finally, the derivatives with respect to the first-layer and second-layer weights are 
given by 


En En 
d ay = jti, = O42). (5.67) 
ðw; Ow;,; 


5.3.3 Efficiency of backpropagation 


One of the most important aspects of backpropagation is its computational effi- 
ciency. To understand this, let us examine how the number of computer operations 
required to evaluate the derivatives of the error function scales with the total number 
W of weights and biases in the network. A single evaluation of the error function 
(for a given input pattern) would require O(W) operations, for sufficiently large W. 
This follows from the fact that, except for a network with very sparse connections, 
the number of weights is typically much greater than the number of units, and so the 
bulk of the computational effort in forward propagation is concerned with evaluat- 
ing the sums in (5.48), with the evaluation of the activation functions representing a 
small overhead. Each term in the sum in (5.48) requires one multiplication and one 
addition, leading to an overall computational cost that is O(W). 

An alternative approach to backpropagation for computing the derivatives of the 
error function is to use finite differences. This can be done by perturbing each weight 
in turn, and approximating the derivatives by the expression 


OEn _ En(wyit¢) ~ Enn) ofe (5.68) 


Ow ji E 


where e < 1. In a software simulation, the accuracy of the approximation to the 
derivatives can be improved by making e smaller, until numerical roundoff problems 
arise. The accuracy of the finite differences method can be improved significantly 
by using symmetrical central differences of the form 

OE, En(Wwji + €) = En (wy: = €) 


= f 2 
a > + O(e?). (5.69) 


In this case, the O(€) corrections cancel, as can be verified by Taylor expansion on 
the right-hand side of (5.69), and so the residual corrections are O(e?). The number 
of computational steps is, however, roughly doubled compared with (5.68). 

The main problem with numerical differentiation is that the highly desirable 
O(W) scaling has been lost. Each forward propagation requires O(W) steps, and 


Figure 5.8 
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Illustration of a modular pattern 

recognition system in which the 

Jacobian matrix can be used v 
to backpropagate error signals 

from the outputs through to ear- 

lier modules in the system. 


there are W weights in the network each of which must be perturbed individually, so 
that the overall scaling is O(W?). 

However, numerical differentiation plays an important role in practice, because a 
comparison of the derivatives calculated by backpropagation with those obtained us- 
ing central differences provides a powerful check on the correctness of any software 
implementation of the backpropagation algorithm. When training networks in prac- 
tice, derivatives should be evaluated using backpropagation, because this gives the 
greatest accuracy and numerical efficiency. However, the results should be compared 
with numerical differentiation using (5.69) for some test cases in order to check the 
correctness of the implementation. 


5.3.4 The Jacobian matrix 


We have seen how the derivatives of an error function with respect to the weights 
can be obtained by the propagation of errors backwards through the network. The 
technique of backpropagation can also be applied to the calculation of other deriva- 
tives. Here we consider the evaluation of the Jacobian matrix, whose elements are 
given by the derivatives of the network outputs with respect to the inputs 


Yk 
Ox; 


Jn = (5.70) 
where each such derivative is evaluated with all other inputs held fixed. Jacobian 
matrices play a useful role in systems built from a number of distinct modules, as 
illustrated in Figure 5.8. Each module can comprise a fixed or adaptive function, 
which can be linear or nonlinear, so long as it is differentiable. Suppose we wish 
to minimize an error function FE with respect to the parameter w in Figure 5.8. The 
derivative of the error function is given by 


OE _ OE Yk 02; 
ðw D Oy, Oz; Ow ean) 
cj 


in which the Jacobian matrix for the red module in Figure 5.8 appears in the middle 
term. 

Because the Jacobian matrix provides a measure of the local sensitivity of the 
outputs to changes in each of the input variables, it also allows any known errors Ax; 
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associated with the inputs to be propagated through the trained network in order to 
estimate their contribution Ay; to the errors at the outputs, through the relation 


Ay. ~ S~ SUK A, (5.72) 


which is valid provided the |Az;| are small. In general, the network mapping rep- 
resented by a trained neural network will be nonlinear, and so the elements of the 
Jacobian matrix will not be constants but will depend on the particular input vector 
used. Thus (5.72) is valid only for small perturbations of the inputs, and the Jacobian 
itself must be re-evaluated for each new input vector. 

The Jacobian matrix can be evaluated using a backpropagation procedure that is 
similar to the one derived earlier for evaluating the derivatives of an error function 
with respect to the weights. We start by writing the element Jg; in the form 


= OYR Oyk Oa; 
Jki an; bD 


Yw Uk (5.73) 
j K 


where we have made use of (5.48). The sum in (5.73) runs over all units j to which 
the input unit ¿ sends connections (for example, over all units in the first hidden 
layer in the layered topology considered earlier). We now write down a recursive 
backpropagation formula to determine the derivatives Oy; /Oa,; 


O"YK yk Oat 
Oa; >, Oa; Oa; 


O"K 
= h'(a;) D5 Fay (5.74) 


where the sum runs over all units / to which unit 7 sends connections (corresponding 
to the first index of wj). Again, we have made use of (5.48) and (5.49). This 
backpropagation starts at the output units for which the required derivatives can be 
found directly from the functional form of the output-unit activation function. For 
instance, if we have individual sigmoidal activation functions at each output unit, 


then 
OYE 


da; = ôkjo' (aj) (5.75) 
whereas for softmax outputs we have 
OYK 
=— = OKjYk — i 5.76 
ða; kj¥k — Ykýj ( ) 


We can summarize the procedure for evaluating the Jacobian matrix as follows. 
Apply the input vector corresponding to the point in input space at which the Ja- 
cobian matrix is to be found, and forward propagate in the usual way to obtain the 
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activations of all of the hidden and output units in the network. Next, for each row 
k of the Jacobian matrix, corresponding to the output unit k, backpropagate using 
the recursive relation (5.74), starting with (5.75) or (5.76), for all of the hidden units 
in the network. Finally, use (5.73) to do the backpropagation to the inputs. The 
Jacobian can also be evaluated using an alternative forward propagation formalism, 
which can be derived in an analogous way to the backpropagation approach given 
here. 

Again, the implementation of such algorithms can be checked by using numeri- 
cal differentiation in the form 


Oye — yR(ti +€) — yr(ai — €) 
Ox; 2e 


Ole) (5.77) 


which involves 2D forward propagations for a network having D inputs. 


The Hessian Matrix 


We have shown how the technique of backpropagation can be used to obtain the first 
derivatives of an error function with respect to the weights in the network. Back- 
propagation can also be used to evaluate the second derivatives of the error, given 
by 
2 
l; (5.78) 
Ow ;;,Ow k 
Note that it is sometimes convenient to consider all of the weight and bias parameters 
as elements w; of a single vector, denoted w, in which case the second derivatives 
form the elements H;; of the Hessian matrix H, where i, j € {1,...,W} and W is 
the total number of weights and biases. The Hessian plays an important role in many 
aspects of neural computing, including the following: 


1. Several nonlinear optimization algorithms used for training neural networks 
are based on considerations of the second-order properties of the error surface, 
which are controlled by the Hessian matrix (Bishop and Nabney, 2008). 


2. The Hessian forms the basis of a fast procedure for re-training a feed-forward 
network following a small change in the training data (Bishop, 1991). 


3. The inverse of the Hessian has been used to identify the least significant weights 
in a network as part of network ‘pruning’ algorithms (Le Cun et al., 1990). 


4. The Hessian plays a central role in the Laplace approximation for a Bayesian 
neural network (see Section 5.7). Its inverse is used to determine the predic- 
tive distribution for a trained network, its eigenvalues determine the values of 
hyperparameters, and its determinant is used to evaluate the model evidence. 


Various approximation schemes have been used to evaluate the Hessian matrix 
for a neural network. However, the Hessian can also be calculated exactly using an 
extension of the backpropagation technique. 
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An important consideration for many applications of the Hessian is the efficiency 
with which it can be evaluated. If there are W parameters (weights and biases) in the 
network, then the Hessian matrix has dimensions W x W and so the computational 
effort needed to evaluate the Hessian will scale like O(W7*) for each pattern in the 
data set. As we shall see, there are efficient methods for evaluating the Hessian 
whose scaling is indeed O(W7?). 


5.4.1 Diagonal approximation 


Some of the applications for the Hessian matrix discussed above require the 
inverse of the Hessian, rather than the Hessian itself. For this reason, there has 
been some interest in using a diagonal approximation to the Hessian, in other words 
one that simply replaces the off-diagonal elements with zeros, because its inverse is 
trivial to evaluate. Again, we shall consider an error function that consists of a sum 
of terms, one for each pattern in the data set, so that E = Oe En. The Hessian can 
then be obtained by considering one pattern at a time, and then summing the results 
over all patterns. From (5.48), the diagonal elements of the Hessian, for pattern n, 
can be written aon op 

z= a 5" 
ðw; Oa; 


Using (5.48) and (5.49), the second derivatives on the right-hand side of (5.79) can 
be found recursively using the chain rule of differential calculus to give a backprop- 
agation equation of the form 


En ., PE 
a h(a) X X wrjwei = T a Caj) Dua ae (5.80) 


kk’ 


(5.79) 


If we now neglect off-diagonal elements in the second-derivative terms, we obtain 
(Becker and Le Cun, 1989; Le Cun et al., 1990) 


OE, 
ða? PEt a 


J 


Tu DEus on (5.81) 


Note that the number of computational steps required to evaluate this approximation 
is O(W), where W is the total number of weight and bias parameters in the network, 
compared with O(W7) for the full Hessian. 

Ricotti et al. (1988) also used the diagonal approximation to the Hessian, but 
they retained all terms in the evaluation of 0? E,, / da‘ and so obtained exact expres- 
sions for the diagonal terms. Note that this no longer has O(W) scaling. The major 
problem with diagonal approximations, however, is that in practice the Hessian is 
typically found to be strongly nondiagonal, and so these approximations, which are 
driven mainly be computational convenience, must be treated with care. 
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5.4.2 Outer product approximation 


When neural networks are applied to regression problems, it is common to use 
a sum-of-squares error function of the form 


B==S (un —- tn)? (5.82) 
n=1 
where we have considered the case of a single output in order to keep the notation 
simple (the extension to several outputs is straightforward). We can then write the 
Hessian matrix in the form 


N N 
H = VVE = X Vyn Vyn + X (Yn — tn) VV Yn- (5.83) 
n=1 n=1 


If the network has been trained on the data set, and its outputs yn happen to be very 
close to the target values t„, then the second term in (5.83) will be small and can 
be neglected. More generally, however, it may be appropriate to neglect this term 
by the following argument. Recall from Section 1.5.5 that the optimal function that 
minimizes a sum-of-squares loss is the conditional average of the target data. The 
quantity (yn — tn) is then a random variable with zero mean. If we assume that its 
value is uncorrelated with the value of the second derivative term on the right-hand 
side of (5.83), then the whole term will average to zero in the summation over n. 

By neglecting the second term in (5.83), we arrive at the Levenberg—Marquardt 
approximation or outer product approximation (because the Hessian matrix is built 
up from a sum of outer products of vectors), given by 


N 
HH >" bab, (5.84) 
n=1 


where b, = Vyn = Van because the activation function for the output units is 
simply the identity. Evaluation of the outer product approximation for the Hessian 
is straightforward as it only involves first derivatives of the error function, which 
can be evaluated efficiently in O(W) steps using standard backpropagation. The 
elements of the matrix can then be found in O(W7) steps by simple multiplication. 
It is important to emphasize that this approximation is only likely to be valid for a 
network that has been trained appropriately, and that for a general network mapping 
the second derivative terms on the right-hand side of (5.83) will typically not be 
negligible. 

In the case of the cross-entropy error function for a network with logistic sigmoid 
output-unit activation functions, the corresponding approximation is given by 


N 
H~ Y yn(1 — gyn) baby: (5.85) 
n=1 


An analogous result can be obtained for multiclass networks having softmax output- 
unit activation functions. 
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5.4.3 Inverse Hessian 


We can use the outer-product approximation to develop a computationally ef- 
ficient procedure for approximating the inverse of the Hessian (Hassibi and Stork, 
1993). First we write the outer-product approximation in matrix notation as 


N 
Hy = ) bab; (5.86) 
n=1 


where bn = Vwan is the contribution to the gradient of the output unit activation 
arising from data point n. We now derive a sequential procedure for building up the 
Hessian by including data points one at a time. Suppose we have already obtained 
the inverse Hessian using the first L data points. By separating off the contribution 
from data point L + 1, we obtain 


Hy41 = Hz + bryibz,). (5.87) 

In order to evaluate the inverse of the Hessian, we now consider the matrix identity 

(M~'v) (v?M~') 
1+v'™M-!v 


where I is the unit matrix, which is simply a special case of the Woodbury identity 
(C.7). If we now identify Hz with M and by; with v, we obtain 
Hz'bz4b}, H7" 


1 +b Hz bry 


(M + vv?) = M`! 


(5.88) 


Ay =H,” 


(5.89) 


In this way, data points are sequentially absorbed until 2+ 1 = N and the whole data 
set has been processed. This result therefore represents a procedure for evaluating 
the inverse of the Hessian using a single pass through the data set. The initial matrix 
Ho is chosen to be aI, where a is a small quantity, so that the algorithm actually 
finds the inverse of H + aI. The results are not particularly sensitive to the precise 
value of a. Extension of this algorithm to networks having more than one output is 
straightforward. 

We note here that the Hessian matrix can sometimes be calculated indirectly as 
part of the network training algorithm. In particular, quasi-Newton nonlinear opti- 
mization algorithms gradually build up an approximation to the inverse of the Hes- 
sian during training. Such algorithms are discussed in detail in Bishop and Nabney 
(2008). 


5.4.4 Finite differences 


As in the case of the first derivatives of the error function, we can find the second 
derivatives by using finite differences, with accuracy limited by numerical precision. 
If we perturb each possible pair of weights in turn, we obtain 


OPE 2 
Owji0wik ~ Ae? 


E(wji — €, wik + €) + E(w;i — €, wik — 2} + O(e?). (5.90) 


{E(wji + €, wik + €) — E(wji + €, wiz — ©) 
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Again, by using a symmetrical central differences formulation, we ensure that the 
residual errors are O(e”) rather than O(¢). Because there are W°? elements in the 
Hessian matrix, and because the evaluation of each element requires four forward 
propagations each needing O(W) operations (per pattern), we see that this approach 
will require O(W*) operations to evaluate the complete Hessian. It therefore has 
poor scaling properties, although in practice it is very useful as a check on the soft- 
ware implementation of backpropagation methods. 

A more efficient version of numerical differentiation can be found by applying 
central differences to the first derivatives of the error function, which are themselves 
calculated using backpropagation. This gives 


OPE 1 OE OE 
= { (wik +€) (wik of + O(e?). (5.91) 


Ow;iOwlk 2€ Ow ji Ow ji 


Because there are now only W weights to be perturbed, and because the gradients 
can be evaluated in O(W) steps, we see that this method gives the Hessian in O(W?) 
operations. 


5.4.5 Exact evaluation of the Hessian 


So far, we have considered various approximation schemes for evaluating the 
Hessian matrix or its inverse. The Hessian can also be evaluated exactly, for a net- 
work of arbitrary feed-forward topology, using extension of the technique of back- 
propagation used to evaluate first derivatives, which shares many of its desirable 
features including computational efficiency (Bishop, 1991; Bishop, 1992). It can be 
applied to any differentiable error function that can be expressed as a function of 
the network outputs and to networks having arbitrary differentiable activation func- 
tions. The number of computational steps needed to evaluate the Hessian scales 
like O(W7). Similar algorithms have also been considered by Buntine and Weigend 
(1993). 

Here we consider the specific case of a network having two layers of weights, 
for which the required equations are easily derived. We shall use indices 7 and 7’ 
to denote inputs, indices j and 7’ to denoted hidden units, and indices k and k’ to 
denote outputs. We first define 


_ OEy OE, 


ôk = Mi = ———_ 
> Ban RR’ = Ga,Oap 


(5.92) 
where En is the contribution to the error from data point n. The Hessian matrix for 
this network can then be considered in three separate blocks as follows. 
1. Both weights in the second layer: 
OE, 


(2) a, (2) 


= zizi Mkk'. (5.93) 
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2. Both weights in the first layer: 


OP Ea D @) 
“aes (a; a) D > wf n . (5.94) 


3. One weight in each layer: 
2E “a = ajh! (aj) $ Onljy + z Y Wry Hew p}. (5.95) 
ðw, Ow 7 


Here I;;, is the j, j’ element of the identity matrix. If one or both of the weights is 
a bias term, then the corresponding expressions are obtained simply by setting the 


Exercise 5.23 appropriate activation(s) to 1. Inclusion of skip-layer connections is straightforward. 


5.4.6 Fast multiplication by the Hessian 


For many applications of the Hessian, the quantity of interest is not the Hessian 
matrix H itself but the product of H with some vector v. We have seen that the 
evaluation of the Hessian takes O(W?) operations, and it also requires storage that is 
O(W?). The vector vTH that we wish to calculate, however, has only W elements, 
so instead of computing the Hessian as an intermediate step, we can instead try to 
find an efficient approach to evaluating v'H directly in a way that requires only 
O(W) operations. 

To do this, we first note that 


v'H=v'V(VE) (5.96) 


where V denotes the gradient operator in weight space. We can then write down 
the standard forward-propagation and backpropagation equations for the evaluation 
of VE and apply (5.96) to these equations to give a set of forward-propagation and 
backpropagation equations for the evaluation of v'H (Møller, 1993; Pearlmutter, 
1994). This corresponds to acting on the original forward-propagation and back- 
propagation equations with a differential operator v' V. Pearlmutter (1994) used the 
notation R{-} to denote the operator vV, and we shall follow this convention. The 
analysis is straightforward and makes use of the usual rules of differential calculus, 
together with the result 

R{w} =v. (5.97) 


The technique is best illustrated with a simple example, and again we choose a 
two-layer network of the form shown in Figure 5.1, with linear output units and a 
sum-of-squares error function. As before, we consider the contribution to the error 
function from one pattern in the data set. The required vector is then obtained as 
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usual by summing over the contributions from each of the patterns separately. For 
the two-layer network, the forward-propagation equations are given by 


a; = 3 On (5.98) 

Zj = h(a;) (5.99) 

ve = >) tga (5.100) 
J 


We now act on these equations using the R{-} operator to obtain a set of forward 
propagation equations in the form 


Rial. = J um (5.101) 

R{z;} h'(aj)R{a;} (5.102) 

Ryk} = So weg {zg} + Y 07923 (5.103) 
j J 


where vj; is the element of the vector v that corresponds to the weight wji. Quan- 
tities of the form R{z;}, R{a;} and R{y,} are to be regarded as new variables 
whose values are found using the above equations. 

Because we are considering a sum-of-squares error function, we have the fol- 
lowing standard backpropagation expressions: 


OK = Yr- tk (5.104) 
5; h'(aj) X weg bk. (5.105) 
k 


Again, we act on these equations with the R{-} operator to obtain a set of backprop- 
agation equations in the form 


R = Rye} (5.106) 
R{O;} A" (aj)R{aj} > Wj Ok 


+ h'(aj) X vrjôr +h! (aj) X > weyR{de}- (5-107) 
k k 


Finally, we have the usual equations for the first derivatives of the error 


OE 

= ip i 
ae 6 (5.108) 
D Sa (5.109) 


Ow ji 
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5.5. 


and acting on these with the R{-} operator, we obtain expressions for the elements 


of the vector v'H 
OE 
Ria) 
OW; 


OF 
ú { Owyi \ 


The implementation of this algorithm involves the introduction of additional 
variables R{a;}, R{z;} and R{ô;} for the hidden units and R{d,} and R{y;} 
for the output units. For each input pattern, the values of these quantities can be 
found using the above results, and the elements of vH are then given by (5.110) 
and (5.111). An elegant aspect of this technique is that the equations for evaluating 
vTH mirror closely those for standard forward and backward propagation, and so the 
extension of existing software to compute this product is typically straightforward. 

If desired, the technique can be used to evaluate the full Hessian matrix by 
choosing the vector v to be given successively by a series of unit vectors of the 
form (0,0,...,1,...,0) each of which picks out one column of the Hessian. This 
leads to a formalism that is analytically equivalent to the backpropagation procedure 
of Bishop (1992), as described in Section 5.4.5, though with some loss of efficiency 
due to redundant calculations. 


R{On} 2; + OnR{z;} (5.110) 


Li R{S;}. (5.111) 


Regularization in Neural Networks 


The number of input and outputs units in a neural network is generally determined 
by the dimensionality of the data set, whereas the number M of hidden units is a free 
parameter that can be adjusted to give the best predictive performance. Note that M 
controls the number of parameters (weights and biases) in the network, and so we 
might expect that in a maximum likelihood setting there will be an optimum value 
of M that gives the best generalization performance, corresponding to the optimum 
balance between under-fitting and over-fitting. Figure 5.9 shows an example of the 
effect of different values of M for the sinusoidal regression problem. 

The generalization error, however, is not a simple function of M due to the 
presence of local minima in the error function, as illustrated in Figure 5.10. Here 
we see the effect of choosing multiple random initializations for the weight vector 
for a range of values of M. The overall best validation set performance in this 
case occurred for a particular solution having M = 8. In practice, one approach to 
choosing M is in fact to plot a graph of the kind shown in Figure 5.10 and then to 
choose the specific solution having the smallest validation set error. 

There are, however, other ways to control the complexity of a neural network 
model in order to avoid over-fitting. From our discussion of polynomial curve fitting 
in Chapter 1, we see that an alternative approach is to choose a relatively large value 
for M and then to control complexity by the addition of a regularization term to the 
error function. The simplest regularizer is the quadratic, giving a regularized error 
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Figure 5.9 Examples of two-layer networks trained on 10 data points drawn from the sinusoidal data set. The 
graphs show the result of fitting networks having M = 1, 3 and 10 hidden units, respectively, by minimizing a 
sum-of-squares error function using a scaled conjugate-gradient algorithm. 


Figure 5.10 


of the form 
E(w) = E(w) + gww. (5.112) 


This regularizer is also known as weight decay and has been discussed at length 
in Chapter 3. The effective model complexity is then determined by the choice of 
the regularization coefficient A. As we have seen previously, this regularizer can be 
interpreted as the negative logarithm of a zero-mean Gaussian prior distribution over 
the weight vector w. 


5.5.1 Consistent Gaussian priors 


One of the limitations of simple weight decay in the form (5.112) is that is 
inconsistent with certain scaling properties of network mappings. To illustrate this, 
consider a multilayer perceptron network having two layers of weights and linear 
output units, which performs a mapping from a set of input variables {2;} to a set 
of output variables {yx}. The activations of the hidden units in the first hidden layer 
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network, with 30 random starts for 
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take the form 


zj=h (= wyits + vn) (5.113) 


u 


while the activations of the output units are given by 


Yk = E Wkjžj + Wko: (5.114) 
j 


Suppose we perform a linear transformation of the input data of the form 
£i > Ti = azi + b. (5.115) 


Then we can arrange for the mapping performed by the network to be unchanged 
by making a corresponding linear transformation of the weights and biases from the 
inputs to the units in the hidden layer of the form 


= 1 
Wji `> Wji = TW ii (5.116) 
a 


: b 
Wj > Üj = wp- Y Uje (5.117) 


Similarly, a linear transformation of the output variables of the network of the form 
Yk Yk = CYk +d (5.118) 


can be achieved by making a transformation of the second-layer weights and biases 
using 


Wkj > Wkj = CWkj (5.119) 
Wko —> Wko = CWro +d. (5.120) 


If we train one network using the original data and one network using data for which 
the input and/or target variables are transformed by one of the above linear transfor- 
mations, then consistency requires that we should obtain equivalent networks that 
differ only by the linear transformation of the weights as given. Any regularizer 
should be consistent with this property, otherwise it arbitrarily favours one solution 
over another, equivalent one. Clearly, simple weight decay (5.112), that treats all 
weights and biases on an equal footing, does not satisfy this property. 

We therefore look for a regularizer which is invariant under the linear trans- 
formations (5.116), (5.117), (5.119) and (5.120). These require that the regularizer 
should be invariant to re-scaling of the weights and to shifts of the biases. Such a 
regularizer is given by 


Ài p A2 2 
a Uka > w (5.121) 
wewy, wEWə 


where W; denotes the set of weights in the first layer, Wz denotes the set of weights 
in the second layer, and biases are excluded from the summations. This regularizer 
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will remain unchanged under the weight transformations provided the regularization 
parameters are re-scaled using Ay — a!/?, and Ap > co !/? Ao. 
The regularizer (5.121) corresponds to a prior of the form 


Q2 2 


Qı 2 
p(w|ai, a2) x exp = > wi > wi). (5.122) 
weEewy, wEWə 


Note that priors of this form are improper (they cannot be normalized) because the 
bias parameters are unconstrained. The use of improper priors can lead to difficulties 
in selecting regularization coefficients and in model comparison within the Bayesian 
framework, because the corresponding evidence is zero. Itis therefore common to 
include separate priors for the biases (which then break shift invariance) having their 
own hyperparameters. We can illustrate the effect of the resulting four hyperpa- 
rameters by drawing samples from the prior and plotting the corresponding network 
functions, as shown in Figure 5.11. 

More generally, we can consider priors in which the weights are divided into 
any number of groups Wọ so that 


1 
p(w) œ exp -32 onllwll (5.123) 
where 
Iwl = $ w. (5.124) 
JEWk 


As a special case of this prior, if we choose the groups to correspond to the sets 
of weights associated with each of the input units, and we optimize the marginal 
likelihood with respect to the corresponding parameters œg, we obtain automatic 
relevance determination as discussed in Section 7.2.2. 


5.5.2 Early stopping 


An alternative to regularization as a way of controlling the effective complexity 
of a network is the procedure of early stopping. The training of nonlinear network 
models corresponds to an iterative reduction of the error function defined with re- 
spect to a set of training data. For many of the optimization algorithms used for 
network training, such as conjugate gradients, the error is a nonincreasing function 
of the iteration index. However, the error measured with respect to independent data, 
generally called a validation set, often shows a decrease at first, followed by an in- 
crease as the network starts to over-fit. Training can therefore be stopped at the point 
of smallest error with respect to the validation data set, as indicated in Figure 5.12, 
in order to obtain a network having good generalization performance. 

The behaviour of the network in this case is sometimes explained qualitatively 
in terms of the effective number of degrees of freedom in the network, in which this 
number starts out small and then to grows during the training process, corresponding 
to a steady increase in the effective complexity of the model. Halting training before 
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Figure 5.11 Illustration of the effect of the hyperparameters governing the prior distribution over weights and 
biases in a two-layer network having a single input, a single linear output, and 12 hidden units having ‘tanh’ 
activation functions. The priors are governed by four hyperparameters a}, a, a3, and a’, which represent 
the precisions of the Gaussian distributions of the first-layer biases, first-layer weights, second-layer biases, and 
second-layer weights, respectively. We see that the parameter a3 governs the vertical scale of functions (note 
the different vertical axis ranges on the top two diagrams), aï governs the horizontal scale of variations in the 
function values, and a? governs the horizontal range over which variations occur. The parameter a}, whose 
effect is not illustrated here, governs the range of vertical offsets of the functions. 
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a minimum of the training error has been reached then represents a way of limiting 
the effective network complexity. 

In the case of a quadratic error function, we can verify this insight, and show 
that early stopping should exhibit similar behaviour to regularization using a sim- 
ple weight-decay term. This can be understood from Figure 5.13, in which the axes 
in weight space have been rotated to be parallel to the eigenvectors of the Hessian 
matrix. If, in the absence of weight decay, the weight vector starts at the origin and 
proceeds during training along a path that follows the local negative gradient vec- 
tor, then the weight vector will move initially parallel to the w2 axis through a point 
corresponding roughly to w and then move towards the minimum of the error func- 
tion wu. This follows from the shape of the error surface and the widely differing 
eigenvalues of the Hessian. Stopping at a point near w is therefore similar to weight 
decay. The relationship between early stopping and weight decay can be made quan- 
titative, thereby showing that the quantity 77 (where 7 is the iteration index, and 7 
is the learning rate parameter) plays the role of the reciprocal of the regularization 
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Figure 5.12 An illustration of the behaviour of training set error (left) and validation set error (right) during a 
typical training session, as a function of the iteration step, for the sinusoidal data set. The goal of achieving 
the best generalization performance suggests that training should be stopped at the point shown by the vertical 
dashed lines, corresponding to the minimum of the validation set error. 


parameter À. The effective number of parameters in the network therefore grows 
during the course of training. 


5.5.3 Invariances 


In many applications of pattern recognition, it is known that predictions should 
be unchanged, or invariant, under one or more transformations of the input vari- 
ables. For example, in the classification of objects in two-dimensional images, such 
as handwritten digits, a particular object should be assigned the same classification 
irrespective of its position within the image (translation invariance) or of its size 
(scale invariance). Such transformations produce significant changes in the raw 
data, expressed in terms of the intensities at each of the pixels in the image, and 
yet should give rise to the same output from the classification system. Similarly 
in speech recognition, small levels of nonlinear warping along the time axis, which 
preserve temporal ordering, should not change the interpretation of the signal. 

If sufficiently large numbers of training patterns are available, then an adaptive 
model such as a neural network can learn the invariance, at least approximately. This 
involves including within the training set a sufficiently large number of examples of 
the effects of the various transformations. Thus, for translation invariance in an im- 
age, the training set should include examples of objects at many different positions. 

This approach may be impractical, however, if the number of training examples 
is limited, or if there are several invariants (because the number of combinations of 
transformations grows exponentially with the number of such transformations). We 
therefore seek alternative approaches for encouraging an adaptive model to exhibit 
the required invariances. These can broadly be divided into four categories: 


1. The training set is augmented using replicas of the training patterns, trans- 
formed according to the desired invariances. For instance, in our digit recog- 
nition example, we could make multiple copies of each example in which the 
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Figure 5.13 A schematic illustration of why w2 


early stopping can give similar 

results to weight decay in the 

case of a quadratic error func- 

tion. The ellipse shows a con- 

tour of constant error, and wu 

denotes the minimum of the er- 

ror function. If the weight vector as 
starts at the origin and moves ac- wW 
cording to the local negative gra- 

dient direction, then it will follow 

the path shown by the curve. By 

stopping training early, a weight 

vector w is found that is qual- w 
itatively similar to that obtained 

with a simple weight-decay reg- 

ularizer and training to the mini- 

mum of the regularized error, as 

can be seen by comparing with 

Figure 3.15. 


digit is shifted to a different position in each image. 


2. A regularization term is added to the error function that penalizes changes in 
the model output when the input is transformed. This leads to the technique of 
tangent propagation, discussed in Section 5.5.4. 


3. Invariance is built into the pre-processing by extracting features that are invari- 
ant under the required transformations. Any subsequent regression or classi- 
fication system that uses such features as inputs will necessarily also respect 
these invariances. 


4. The final option is to build the invariance properties into the structure of a neu- 
ral network (or into the definition of a kernel function in the case of techniques 
such as the relevance vector machine). One way to achieve this is through the 
use of local receptive fields and shared weights, as discussed in the context of 
convolutional neural networks in Section 5.5.6. 


Approach 1 is often relatively easy to implement and can be used to encourage com- 
plex invariances such as those illustrated in Figure 5.14. For sequential training 
algorithms, this can be done by transforming each input pattern before it is presented 
to the model so that, if the patterns are being recycled, a different transformation 
(drawn from an appropriate distribution) is added each time. For batch methods, a 
similar effect can be achieved by replicating each data point a number of times and 
transforming each copy independently. The use of such augmented data can lead to 
significant improvements in generalization (Simard et al., 2003), although it can also 
be computationally costly. 

Approach 2 leaves the data set unchanged but modifies the error function through 
the addition of a regularizer. In Section 5.5.5, we shall show that this approach is 
closely related to approach 2. 
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Figure 5.14 Illustration of the synthetic warping of a handwritten digit. The original image is shown on the 
left. On the right, the top row shows three examples of warped digits, with the corresponding displacement 
fields shown on the bottom row. These displacement fields are generated by sampling random displacements 
Az, Ay € (0,1) at each pixel and then smoothing by convolution with Gaussians of width 0.01, 30 and 60 


respectively. 


Figure 5.15 


One advantage of approach 3 is that it can correctly extrapolate well beyond the 
range of transformations included in the training set. However, it can be difficult 
to find hand-crafted features with the required invariances that do not also discard 
information that can be useful for discrimination. 


5.5.4 Tangent propagation 


We can use regularization to encourage models to be invariant to transformations 
of the input through the technique of tangent propagation (Simard et al., 1992). 
Consider the effect of a transformation on a particular input vector x,,. Provided the 
transformation is continuous (such as translation or rotation, but not mirror reflection 
for instance), then the transformed pattern will sweep out a manifold M within the 
D-dimensional input space. This is illustrated in Figure 5.15, for the case of D = 
2 for simplicity. Suppose the transformation is governed by a single parameter € 
(which might be rotation angle for instance). Then the subspace M swept out by x, 


Illustration of a two-dimensional input space %2 
showing the effect of a continuous transforma- 
tion on a particular input vector xn. A one- 
dimensional transformation, parameterized by 
the continuous variable €, applied to xn causes 
it to sweep out a one-dimensional manifold M. 
Locally, the effect of the transformation can be 
approximated by the tangent vector Tn. 
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will be one-dimensional, and will be parameterized by £. Let the vector that results 
from acting on x,, by this transformation be denoted by s(x,,,€), which is defined 
so that s(x,0) = x. Then the tangent to the curve M is given by the directional 
derivative 7 = Os/O€, and the tangent vector at the point x,, is given by 


Os(Xn,§) 


BE (5.125) 


ran MnO) 
€=0 


Under a transformation of the input vector, the network output vector will, in general, 
change. The derivative of output k with respect to € is given by 


_ = OyK Ox; 
fl Ox; oE 


OYE 
Og 


D 
= > Just (5.126) 
£=0 i=1 


€=0 


where Jp: is the (k, i) element of the Jacobian matrix J, as discussed in Section 5.3.4. 
The result (5.126) can be used to modify the standard error function, so as to encour- 
age local invariance in the neighbourhood of the data points, by the addition to the 
original error function E of a regularization function Q to give a total error function 
of the form a 

B=E+ AQ (5.127) 


where is a regularization coefficient and 


2 D 2 
1 OYn 1 


The regularization function will be zero when the network mapping function is in- 
variant under the transformation in the neighbourhood of each pattern vector, and 
the value of the parameter A determines the balance between fitting the training data 
and learning the invariance property. 

In a practical implementation, the tangent vector Tn can be approximated us- 
ing finite differences, by subtracting the original vector x,, from the corresponding 
vector after transformation using a small value of £, and then dividing by £. This is 
illustrated in Figure 5.16. 

The regularization function depends on the network weights through the Jaco- 
bian J. A backpropagation formalism for computing the derivatives of the regu- 
larizer with respect to the network weights is easily obtained by extension of the 
techniques introduced in Section 5.3. 

If the transformation is governed by L parameters (e.g., L = 3 for the case of 
translations combined with in-plane rotations in a two-dimensional image), then the 
manifold M will have dimensionality L, and the corresponding regularizer is given 
by the sum of terms of the form (5.128), one for each transformation. If several 
transformations are considered at the same time, and the network mapping is made 
invariant to each separately, then it will be (locally) invariant to combinations of the 
transformations (Simard et al., 1992). 


Figure 5.16 Illustration showing 
(a) the original image x of a hand- 
written digit, (b) the tangent vector 
T corresponding to an infinitesimal 
clockwise rotation, (c) the result of 
adding a small contribution from the 
tangent vector to the original image 
giving x + er with e = 15 degrees, 
and (d) the true image rotated for 
comparison. 
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- 


(c) (d) 


A related technique, called tangent distance, can be used to build invariance 
properties into distance-based methods such as nearest-neighbour classifiers (Simard 
et al., 1993). 


5.5.5 Training with transformed data 


We have seen that one way to encourage invariance of a model to a set of trans- 
formations is to expand the training set using transformed versions of the original 
input patterns. Here we show that this approach is closely related to the technique of 
tangent propagation (Bishop, 1995b; Leen, 1995). 

As in Section 5.5.4, we shall consider a transformation governed by a single 
parameter € and described by the function s(x,€), with s(x,0) = x. We shall 
also consider a sum-of-squares error function. The error function for untransformed 
inputs can be written (in the infinite data set limit) in the form 


E = al {y(x) — t}*p(t|x)p(x) dx dt (5.129) 


as discussed in Section 1.5.5. Here we have considered a network having a single 
output, in order to keep the notation uncluttered. If we now consider an infinite 
number of copies of each data point, each of which is perturbed by the transformation 


266 


5. NEURAL NETWORKS 


in which the parameter € is drawn from a distribution p(¢), then the error function 
defined over this expanded data set can be written as 


B= ; i | {y(s(x, €)) — t}2p(tlx)p(x)p(é) dxdt dé. (5.130) 


We now assume that the distribution p(€) has zero mean with small variance, so that 
we are only considering small transformations of the original input vectors. We can 
then expand the transformation function as a Taylor series in powers of € to give 


ð 
a£ 


s£) + se) + O(€*) 
3 


, = ;0 + 
s(x, £) s(x,0)+€ 2 o 


X+HET + ler + O(€?) 


where T’ denotes the second derivative of s(x, £) with respect to € evaluated at € = 0. 
This allows us to expand the model function to give 


ul s(x) = ul) + €r? ule) + È (ay Vy) +7" VV ye] + OC). 


Substituting into the mean error function (5.130) and expanding, we then have 
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+ Eg I =i mtd 
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+ ("Vy ))* p(t|x)p(x) dx dt + O(€°). 


Because the distribution of transformations has zero mean we have E|] = 0. Also, 
we shall denote E[£*] by A. Omitting terms of O(€*), the average error function then 
becomes 


E=E+)0 (5.131) 


where F is the original sum-of-squares error, and the regularization term Q takes the 
form 


Q = f ju = ESIE f" Vy(x) + TT VVy(x)r} 
+ (rvv) p(x) dx (5.132) 


in which we have performed the integration over t. 
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We can further simplify this regularization term as follows. In Section 1.5.5 we 
saw that the function that minimizes the sum-of-squares error is given by the condi- 
tional average E|t|x] of the target values t. From (5.131) we see that the regularized 
error will equal the unregularized sum-of-squares plus terms which are O(£), and so 
the network function that minimizes the total error will have the form 


y(x) = E[t|x] + O(€). (5.133) 


Thus, to leading order in £, the first term in the regularizer vanishes and we are left 
with 


Q= al (tT Vy(x))” p(x) dx (5.134) 


which is equivalent to the tangent propagation regularizer (5.128). 

If we consider the special case in which the transformation of the inputs simply 
consists of the addition of random noise, so that x — x + &, then the regularizer 
takes the form 


1 
= 5 J IPoP) dx (5.135) 


which is known as Tikhonov regularization (Tikhonov and Arsenin, 1977; Bishop, 
1995b). Derivatives of this regularizer with respect to the network weights can be 
found using an extended backpropagation algorithm (Bishop, 1993). We see that, for 
small noise amplitudes, Tikhonov regularization is related to the addition of random 
noise to the inputs, which has been shown to improve generalization in appropriate 
circumstances (Sietsma and Dow, 1991). 


5.5.6 Convolutional networks 


Another approach to creating models that are invariant to certain transformation 
of the inputs is to build the invariance properties into the structure of a neural net- 
work. This is the basis for the convolutional neural network (Le Cun et al., 1989; 
LeCun et al., 1998), which has been widely applied to image data. 

Consider the specific task of recognizing handwritten digits. Each input image 
comprises a set of pixel intensity values, and the desired output is a posterior proba- 
bility distribution over the ten digit classes. We know that the identity of the digit is 
invariant under translations and scaling as well as (small) rotations. Furthermore, the 
network must also exhibit invariance to more subtle transformations such as elastic 
deformations of the kind illustrated in Figure 5.14. One simple approach would be to 
treat the image as the input to a fully connected network, such as the kind shown in 
Figure 5.1. Given a sufficiently large training set, such a network could in principle 
yield a good solution to this problem and would learn the appropriate invariances by 
example. 

However, this approach ignores a key property of images, which is that nearby 
pixels are more strongly correlated than more distant pixels. Many of the modern 
approaches to computer vision exploit this property by extracting local features that 
depend only on small subregions of the image. Information from such features can 
then be merged in later stages of processing in order to detect higher-order features 
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Sub-sampling 
layer 


Input image Convolutional layer 


Figure 5.17 Diagram illustrating part of a convolutional neural network, showing a layer of convolu- 


tional units followed by a layer of subsampling units. Several successive pairs of such 
layers may be used. 


and ultimately to yield information about the image as whole. Also, local features 
that are useful in one region of the image are likely to be useful in other regions of 
the image, for instance if the object of interest is translated. 

These notions are incorporated into convolutional neural networks through three 
mechanisms: (i) local receptive fields, (ii) weight sharing, and (iii) subsampling. The 
structure of a convolutional network is illustrated in Figure 5.17. In the convolutional 
layer the units are organized into planes, each of which is called a feature map. Units 
in a feature map each take inputs only from a small subregion of the image, and all 
of the units in a feature map are constrained to share the same weight values. For 
instance, a feature map might consist of 100 units arranged in a 10 x 10 grid, with 
each unit taking inputs from a 5 x 5 pixel patch of the image. The whole feature map 
therefore has 25 adjustable weight parameters plus one adjustable bias parameter. 
Input values from a patch are linearly combined using the weights and the bias, and 
the result transformed by a sigmoidal nonlinearity using (5.1). If we think of the units 
as feature detectors, then all of the units in a feature map detect the same pattern but 
at different locations in the input image. Due to the weight sharing, the evaluation 
of the activations of these units is equivalent to a convolution of the image pixel 
intensities with a ‘kernel’ comprising the weight parameters. If the input image is 
shifted, the activations of the feature map will be shifted by the same amount but will 
otherwise be unchanged. This provides the basis for the (approximate) invariance of 
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the network outputs to translations and distortions of the input image. Because we 
will typically need to detect multiple features in order to build an effective model, 
there will generally be multiple feature maps in the convolutional layer, each having 
its own set of weight and bias parameters. 

The outputs of the convolutional units form the inputs to the subsampling layer 
of the network. For each feature map in the convolutional layer, there is a plane of 
units in the subsampling layer and each unit takes inputs from a small receptive field 
in the corresponding feature map of the convolutional layer. These units perform 
subsampling. For instance, each subsampling unit might take inputs from a 2 x 2 
unit region in the corresponding feature map and would compute the average of 
those inputs, multiplied by an adaptive weight with the addition of an adaptive bias 
parameter, and then transformed using a sigmoidal nonlinear activation function. 
The receptive fields are chosen to be contiguous and nonoverlapping so that there 
are half the number of rows and columns in the subsampling layer compared with 
the convolutional layer. In this way, the response of a unit in the subsampling layer 
will be relatively insensitive to small shifts of the image in the corresponding regions 
of the input space. 

In a practical architecture, there may be several pairs of convolutional and sub- 
sampling layers. At each stage there is a larger degree of invariance to input trans- 
formations compared to the previous layer. There may be several feature maps in a 
given convolutional layer for each plane of units in the previous subsampling layer, 
so that the gradual reduction in spatial resolution is then compensated by an increas- 
ing number of features. The final layer of the network would typically be a fully 
connected, fully adaptive layer, with a softmax output nonlinearity in the case of 
multiclass classification. 

The whole network can be trained by error minimization using backpropagation 
to evaluate the gradient of the error function. This involves a slight modification 
of the usual backpropagation algorithm to ensure that the shared-weight constraints 
are satisfied. Due to the use of local receptive fields, the number of weights in 
the network is smaller than if the network were fully connected. Furthermore, the 
number of independent parameters to be learned from the data is much smaller still, 
due to the substantial numbers of constraints on the weights. 


5.5.7 Soft weight sharing 


One way to reduce the effective complexity of a network with a large number 
of weights is to constrain weights within certain groups to be equal. This is the 
technique of weight sharing that was discussed in Section 5.5.6 as a way of building 
translation invariance into networks used for image interpretation. It is only appli- 
cable, however, to particular problems in which the form of the constraints can be 
specified in advance. Here we consider a form of soft weight sharing (Nowlan and 
Hinton, 1992) in which the hard constraint of equal weights is replaced by a form 
of regularization in which groups of weights are encouraged to have similar values. 
Furthermore, the division of weights into groups, the mean weight value for each 
group, and the spread of values within the groups are all determined as part of the 
learning process. 
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Recall that the simple weight decay regularizer, given in (5.112), can be viewed 
as the negative log of a Gaussian prior distribution over the weights. We can encour- 
age the weight values to form several groups, rather than just one group, by consid- 
ering instead a probability distribution that is a mixture of Gaussians. The centres 
and variances of the Gaussian components, as well as the mixing coefficients, will be 
considered as adjustable parameters to be determined as part of the learning process. 
Thus, we have a probability density of the form 


= | [ 2w) (5.136) 


where 
M 


p(wi) = X miN (wily, o3) (5.137) 


and 7; are the mixing coefficients. Taking the negative logarithm then leads to a 
regularization function of the form 


M 
==- h 2 TiN (wales, a») . (5.138) 
i j=1 
The total error function is then given by 


E(w) = E(w) + AQ(w) (5.139) 


where A is the regularization coefficient. This error is minimized both with respect 
to the weights w; and with respect to the parameters {7;, 4;,0;} of the mixture 
model. If the weights were constant, then the parameters of the mixture model could 
be determined by using the EM algorithm discussed in Chapter 9. However, the dis- 
tribution of weights is itself evolving during the learning process, and so to avoid nu- 
merical instability, a joint optimization is performed simultaneously over the weights 
and the mixture-model parameters. This can be done using a standard optimization 
algorithm such as conjugate gradients or quasi- Newton methods. 

In order to minimize the total error function, it is necessary to be able to evaluate 
its derivatives with respect to the various adjustable parameters. To do this it is con- 
venient to regard the {7} as prior probabilities and to introduce the corresponding 
posterior probabilities which, following (2.192), are given by Bayes’ theorem in the 
form 

TiN (wl tj, 03) 


WN =e Ail, a 


The derivatives of the total error function with respect to the weights are then given 
by 


(5.140) 
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The effect of the regularization term is therefore to pull each weight towards the 
centre of the j*" Gaussian, with a force proportional to the posterior probability of 
that Gaussian for the given weight. This is precisely the kind of effect that we are 
seeking. 

Derivatives of the error with respect to the centres of the Gaussians are also 
easily computed to give 


BF Dw) A (5.142) 
Hj 7 
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which has a simple intuitive interpretation, because it pushes uj towards an aver- 
age of the weight values, weighted by the posterior probabilities that the respective 
weight parameters were generated by component j. Similarly, the derivatives with 
respect to the variances are given by 


JE 1 (ty p)? 
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which drives g; towards the weighted average of the squared deviations of the weights 
around the corresponding centre uj, where the weighting coefficients are again given 
by the posterior probability that each weight is generated by component 7. Note that 
in a practical implementation, new variables 7; defined by 


o? = exp(1;) (5.144) 


are introduced, and the minimization is performed with respect to the 7;. This en- 
sures that the parameters 0; remain positive. It also has the effect of discouraging 
pathological solutions in which one or more of the 7; goes to zero, corresponding 
to a Gaussian component collapsing onto one of the weight parameter values. Such 
solutions are discussed in more detail in the context of Gaussian mixture models in 
Section 9.2.1. 

For the derivatives with respect to the mixing coefficients 7;, we need to take 
account of the constraints 


SG) O<m<1 (5.145) 
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which follow from the interpretation of the 7; as prior probabilities. This can be 
done by expressing the mixing coefficients in terms of a set of auxiliary variables 
{n; } using the softmax function given by 


erm) 
Spaa exp (ne) 


The derivatives of the regularized error function with respect to the {7;} then take 
the form 


(5.146) 
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Figure 5.18 The left figure shows a two-link robot arm, 
in which the Cartesian coordinates (21, x2) of the end ef- 
fector are determined uniquely by the two joint angles 0; 
and 62 and the (fixed) lengths Lı and Lə of the arms. This 
is know as the forward kinematics of the arm. In prac- 
tice, we have to find the joint angles that will give rise to a 
desired end effector position and, as shown in the right fig- 
ure, this inverse kinematics has two solutions correspond- 
ing to ‘elbow up’ and ‘elbow down’. 
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(5.147) 


We see that 7; is therefore driven towards the average posterior probability for com- 
ponent 7. 


Mixture Density Networks 


The goal of supervised learning is to model a conditional distribution p(t|x), which 
for many simple regression problems is chosen to be Gaussian. However, practical 
machine learning problems can often have significantly non-Gaussian distributions. 
These can arise, for example, with inverse problems in which the distribution can be 
multimodal, in which case the Gaussian assumption can lead to very poor predic- 
tions. 

As a simple example of an inverse problem, consider the kinematics of a robot 
arm, as illustrated in Figure 5.18. The forward problem involves finding the end ef- 
fector position given the joint angles and has a unique solution. However, in practice 
we wish to move the end effector of the robot to a specific position, and to do this we 
must set appropriate joint angles. We therefore need to solve the inverse problem, 
which has two solutions as seen in Figure 5.18. 

Forward problems often corresponds to causality in a physical system and gen- 
erally have a unique solution. For instance, a specific pattern of symptoms in the 
human body may be caused by the presence of a particular disease. In pattern recog- 
nition, however, we typically have to solve an inverse problem, such as trying to 
predict the presence of a disease given a set of symptoms. If the forward problem 
involves a many-to-one mapping, then the inverse problem will have multiple solu- 
tions. For instance, several different diseases may result in the same symptoms. 

In the robotics example, the kinematics is defined by geometrical equations, and 
the multimodality is readily apparent. However, in many machine learning problems 
the presence of multimodality, particularly in problems involving spaces of high di- 
mensionality, can be less obvious. For tutorial purposes, however, we shall consider 
a simple toy problem for which we can easily visualize the multimodality. Data for 
this problem is generated by sampling a variable x uniformly over the interval (0, 1), 
to give a set of values {£n}, and the corresponding target values t,, are obtained 
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Figure 5.19 On the left is the data 
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set for a simple ‘forward problem’ in 1 
which the red curve shows the result 
of fitting a two-layer neural network 
by minimizing the sum-of-squares 
error function. The corresponding 
inverse problem, shown on the right, 
is obtained by exchanging the roles 
of x and t. Here the same net- 
work trained again by minimizing the 
sum-of-squares error function gives 
a very poor fit to the data due to the O 


multimodality of the data set. 


by computing the function £„ + 0.3 sin(27,,) and then adding uniform noise over 
the interval (—0.1,0.1). The inverse problem is then obtained by keeping the same 
data points but exchanging the roles of x and t. Figure 5.19 shows the data sets for 
the forward and inverse problems, along with the results of fitting two-layer neural 
networks having 6 hidden units and a single linear output unit by minimizing a sum- 
of-squares error function. Least squares corresponds to maximum likelihood under 
a Gaussian assumption. We see that this leads to a very poor model for the highly 
non-Gaussian inverse problem. 

We therefore seek a general framework for modelling conditional probability 
distributions. This can be achieved by using a mixture model for p(t|x) in which 
both the mixing coefficients as well as the component densities are flexible functions 
of the input vector x, giving rise to the mixture density network. For any given value 
of x, the mixture model provides a general formalism for modelling an arbitrary 
conditional density function p(t|x). Provided we consider a sufficiently flexible 
network, we then have a framework for approximating arbitrary conditional distri- 
butions. 

Here we shall develop the model explicitly for Gaussian components, so that 


p(t|x) = Sms N (t| 4 (x), o2(x)) - (5.148) 


This is an example of a heteroscedastic model since the noise variance on the data 
is a function of the input vector x. Instead of Gaussians, we can use other distribu- 
tions for the components, such as Bernoulli distributions if the target variables are 
binary rather than continuous. We have also specialized to the case of isotropic co- 
variances for the components, although the mixture density network can readily be 
extended to allow for general covariance matrices by representing the covariances 
using a Cholesky factorization (Williams, 1996). Even with isotropic components, 
the conditional distribution p(t|x) does not assume factorization with respect to the 
components of t (in contrast to the standard sum-of-squares regression model) as a 
consequence of the mixture distribution. 

We now take the various parameters of the mixture model, namely the mixing 
coefficients 7;,(x), the means p;,(x), and the variances o7(x), to be governed by 
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t 


Figure 5.20 The mixture density network can represent general conditional probability densities p(t|x) 


by considering a parametric mixture model for the distribution of t whose parameters are 
determined by the outputs of a neural network that takes x as its input vector. 


the outputs of a conventional neural network that takes x as its input. The structure 
of this mixture density network is illustrated in Figure 5.20. The mixture density 
network is closely related to the mixture of experts discussed in Section 14.5.3. The 
principle difference is that in the mixture density network the same function is used 
to predict the parameters of all of the component densities as well as the mixing co- 
efficients, and so the nonlinear hidden units are shared amongst the input-dependent 
functions. 

The neural network in Figure 5.20 can, for example, be a two-layer network 
having sigmoidal (‘tanh’) hidden units. If there are L components in the mixture 
model (5.148), and if t has K components, then the network will have L output unit 
activations denoted by aj that determine the mixing coefficients m(x), K outputs 
denoted by af that determine the kernel widths o;,(x), and L x K outputs denoted 
by ak j that determine the components 1x; (x) of the kernel centres j2;,(x). The total 
number of network outputs is given by (K + 2)L, as compared with the usual K 
outputs for a network, which simply predicts the conditional means of the target 
variables. 

The mixing coefficients must satisfy the constraints 


K 
S oml) =1, 0 < m(x) <1 (5.149) 
k=1 


which can be achieved using a set of softmax outputs 


exp(az) 
K aA 
Xia exp(a7) 


Similarly, the variances must satisfy o7(x) > 0 and so can be represented in terms 
of the exponentials of the corresponding network activations using 


m(x) = (5.150) 


on(x) = exp(az). (5.151) 


Finally, because the means u(x) have real components, they can be represented 
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directly by the network output activations 
Hkj(x) = akj (5.152) 


The adaptive parameters of the mixture density network comprise the vector w 
of weights and biases in the neural network, that can be set by maximum likelihood, 
or equivalently by minimizing an error function defined to be the negative logarithm 
of the likelihood. For independent data, this error function takes the form 


N k 
E(w)=-— Xon 5 TklXn, WIN (trl tg (nsw), cto.) (5.153) 
k=1 


n=l 


where we have made the dependencies on w explicit. 

In order to minimize the error function, we need to calculate the derivatives of 
the error E(w) with respect to the components of w. These can be evaluated by 
using the standard backpropagation procedure, provided we obtain suitable expres- 
sions for the derivatives of the error with respect to the output-unit activations. These 
represent error signals ô for each pattern and for each output unit, and can be back- 
propagated to the hidden units and the error function derivatives evaluated in the 
usual way. Because the error function (5.153) is composed of a sum of terms, one 
for each training data point, we can consider the derivatives for a particular pattern 
n and then find the derivatives of E by summing over all patterns. 

Because we are dealing with mixture distributions, it is convenient to view the 
mixing coefficients m(x) as x-dependent prior probabilities and to introduce the 
corresponding posterior probabilities given by 


TN nk 
La TNn 
where Nng denotes N (tn | Hp (Xn), 72(Xn))- 


The derivatives with respect to the network output activations governing the mix- 
ing coefficients are given by 


w(t|x) = (5.154) 


En 
= Tk — Yk. (5.155) 
da7. 
Similarly, the derivatives with respect to the output activations controlling the com- 
ponent means are given by 
OE, Mei — tı 
=k : 5.156 
ma = a = 


Finally, the derivatives with respect to the output activations controlling the compo- 
nent variances are given by 


E t= 2 1 


af, o} Ok 


276 5. NEURAL NETWORKS 


Figure 5.21 (a) Plot of the mixing 
coefficients m(x) as a function of 1 1f 
x for the three kernel functions in a 
mixture density network trained on 
the data shown in Figure 5.19. The 
model has three Gaussian compo- 
nents, and uses a two-layer multi- 
layer perceptron with five ‘tanh’ sig- 
moidal units in the hidden layer, and 
nine outputs (corresponding to the 3 
means and 3 variances of the Gaus- 


sian components and the 3 mixing o 1 0 
coefficients). At both small and large 

values of x, where the conditional (a) 

probability density of the target data 
is unimodal, only one of the ker- 


nels has a high value for its prior 1 
probability, while at intermediate val- 
ues of x, where the conditional den- 
sity is trimodal, the three mixing co- 
efficients have comparable values. 
(b) Plots of the means u(x) using 
the same colour coding as for the 
mixing coefficients. (c) Plot of the 
contours of the corresponding con- 
ditional probability density of the tar- 0 


get data for the same mixture den- 0 1 0 
sity network. (d) Plot of the ap- 

proximate conditional mode, shown (c) 

by the red points, of the conditional 

density. 


We illustrate the use of a mixture density network by returning to the toy ex- 


ample of an inverse problem shown in Figure 5.19. Plots of the mixing coeffi- 
cients m(x), the means u(x), and the conditional density contours corresponding 
to p(t|x), are shown in Figure 5.21. The outputs of the neural network, and hence the 
parameters in the mixture model, are necessarily continuous single-valued functions 
of the input variables. However, we see from Figure 5.21(c) that the model is able to 
produce a conditional density that is unimodal for some values of x and trimodal for 
other values by modulating the amplitudes of the mixing components 7;,(x). 

Once a mixture density network has been trained, it can predict the conditional 
density function of the target data for any given value of the input vector. This 
conditional density represents a complete description of the generator of the data, so 
far as the problem of predicting the value of the output vector is concerned. From 
this density function we can calculate more specific quantities that may be of interest 
in different applications. One of the simplest of these is the mean, corresponding to 
the conditional average of the target data, and is given by 


K 
E [t|x] = / tp(t|x) dt = X` ma (x) py (x) (5.158) 
k=1 
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where we have used (5.148). Because a standard network trained by least squares 
is approximating the conditional mean, we see that a mixture density network can 
reproduce the conventional least-squares result as a special case. Of course, as we 
have already noted, for a multimodal distribution the conditional mean is of limited 
value. 

We can similarly evaluate the variance of the density function about the condi- 
tional average, to give 


s(x) = E||\t —E[t|x]|| [x] (5.159) 


K 
= YO m(x) § 02 (x) + fe) -Y mem] $ 6.160 
k=l 


l=1 


where we have used (5.148) and (5.158). This is more general than the corresponding 
least-squares result because the variance is a function of x. 

We have seen that for multimodal distributions, the conditional mean can give 
a poor representation of the data. For instance, in controlling the simple robot arm 
shown in Figure 5.18, we need to pick one of the two possible joint angle settings 
in order to achieve the desired end-effector location, whereas the average of the two 
solutions is not itself a solution. In such cases, the conditional mode may be of 
more value. Because the conditional mode for the mixture density network does not 
have a simple analytical solution, this would require numerical iteration. A simple 
alternative is to take the mean of the most probable component (i.e., the one with the 
largest mixing coefficient) at each value of x. This is shown for the toy data set in 
Figure 5.21 (d). 


Bayesian Neural Networks 


So far, our discussion of neural networks has focussed on the use of maximum like- 
lihood to determine the network parameters (weights and biases). Regularized max- 
imum likelihood can be interpreted as a MAP (maximum posterior) approach in 
which the regularizer can be viewed as the logarithm of a prior parameter distribu- 
tion. However, in a Bayesian treatment we need to marginalize over the distribution 
of parameters in order to make predictions. 

In Section 3.3, we developed a Bayesian solution for a simple linear regression 
model under the assumption of Gaussian noise. We saw that the posterior distribu- 
tion, which is Gaussian, could be evaluated exactly and that the predictive distribu- 
tion could also be found in closed form. In the case of a multilayered network, the 
highly nonlinear dependence of the network function on the parameter values means 
that an exact Bayesian treatment can no longer be found. In fact, the log of the pos- 
terior distribution will be nonconvex, corresponding to the multiple local minima in 
the error function. 

The technique of variational inference, to be discussed in Chapter 10, has been 
applied to Bayesian neural networks using a factorized Gaussian approximation 
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to the posterior distribution (Hinton and van Camp, 1993) and also using a full- 
covariance Gaussian (Barber and Bishop, 1998a; Barber and Bishop, 1998b). The 
most complete treatment, however, has been based on the Laplace approximation 
(MacKay, 1992c; MacKay, 1992b) and forms the basis for the discussion given here. 
We will approximate the posterior distribution by a Gaussian, centred at a mode of 
the true posterior. Furthermore, we shall assume that the covariance of this Gaus- 
sian is small so that the network function is approximately linear with respect to the 
parameters over the region of parameter space for which the posterior probability is 
significantly nonzero. With these two approximations, we will obtain models that 
are analogous to the linear regression and classification models discussed in earlier 
chapters and so we can exploit the results obtained there. We can then make use of 
the evidence framework to provide point estimates for the hyperparameters and to 
compare alternative models (for example, networks having different numbers of hid- 
den units). To start with, we shall discuss the regression case and then later consider 
the modifications needed for solving classification tasks. 


5.7.1 Posterior parameter distribution 


Consider the problem of predicting a single continuous target variable t from 
a vector x of inputs (the extension to multiple targets is straightforward). We shall 
suppose that the conditional distribution p(t|x) is Gaussian, with an x-dependent 
mean given by the output of a neural network model y(x, w), and with precision 
(inverse variance) 8 


p(t|x, w, B) = N(tly(x, w), 87+). (5.161) 


Similarly, we shall choose a prior distribution over the weights w that is Gaussian of 
the form 
p(wla) = N(w|0,a7'T). (5.162) 


For an i.i.d. data set of N observations x,,..., Xy, with a corresponding set of target 
values D = {t,,...,ty}, the likelihood function is given by 


N 


p(D|w, p) = [Glyn w), 07) (5.163) 


n=1 
and so the resulting posterior distribution is then 
p(w|D, a, 8) x p(wla)p(D|w, 8). (5.164) 


which, as a consequence of the nonlinear dependence of y(x, w) on w, will be non- 
Gaussian. 

We can find a Gaussian approximation to the posterior distribution by using the 
Laplace approximation. To do this, we must first find a (local) maximum of the 
posterior, and this must be done using iterative numerical optimization. As usual, it 
is convenient to maximize the logarithm of the posterior, which can be written in the 
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form 
a B a 
= T 2 
ln p(w|D) = “i We 3 {y(Xn, W) — tn} + const (5.165) 


which corresponds to a regularized sum-of-squares error function. Assuming for 
the moment that a and 8 are fixed, we can find a maximum of the posterior, which 
we denote wmap, by standard nonlinear optimization algorithms such as conjugate 
gradients, using error backpropagation to evaluate the required derivatives. 

Having found a mode wmap, we can then build a local Gaussian approximation 
by evaluating the matrix of second derivatives of the negative log posterior distribu- 
tion. From (5.165), this is given by 


A = -VV ln p(w|D, a, 3) = al + GH (5.166) 


where H is the Hessian matrix comprising the second derivatives of the sum-of- 
squares error function with respect to the components of w. Algorithms for comput- 
ing and approximating the Hessian were discussed in Section 5.4. The corresponding 
Gaussian approximation to the posterior is then given from (4.134) by 


q(w|D) = N(w|wmap, A’). (5.167) 


Similarly, the predictive distribution is obtained by marginalizing with respect 
to this posterior distribution 


p(t|x, D) = J viis wawi) dw. (5.168) 


However, even with the Gaussian approximation to the posterior, this integration is 
still analytically intractable due to the nonlinearity of the network function y(x, w) 
as a function of w. To make progress, we now assume that the posterior distribution 
has small variance compared with the characteristic scales of w over which y(x, w) 
is varying. This allows us to make a Taylor series expansion of the network function 
around wap and retain only the linear terms 


y(x, w) © y(x, wuap) + g* (W — Wap) (5.169) 


where we have defined 
g = VwY(X,W)lwowsrap ’ (5.170) 


With this approximation, we now have a linear-Gaussian model with a Gaussian 
distribution for p(w) and a Gaussian for p(t|w) whose mean is a linear function of 
w of the form 


p(t|x, W, B) = N (tly (x, WMAP) + gT(w 7 WMAP), ge) . (5.171) 
We can therefore make use of the general result (2.115) for the marginal p(t) to give 


p(t|x, D, a, 8) =N (tly(x, wap), o° (x)) (5.172) 
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where the input-dependent variance is given by 
o*(x) = 6 '+g7A'g. (5.173) 


We see that the predictive distribution p(t|x, D) is a Gaussian whose mean is given 
by the network function y(x, wap) with the parameter set to their MAP value. The 
variance has two terms, the first of which arises from the intrinsic noise on the target 
variable, whereas the second is an x-dependent term that expresses the uncertainty 
in the interpolant due to the uncertainty in the model parameters w. This should 
be compared with the corresponding predictive distribution for the linear regression 
model, given by (3.58) and (3.59). 


5.7.2 Hyperparameter optimization 


So far, we have assumed that the hyperparameters a and £ are fixed and known. 
We can make use of the evidence framework, discussed in Section 3.5, together with 
the Gaussian approximation to the posterior obtained using the Laplace approxima- 
tion, to obtain a practical procedure for choosing the values of such hyperparameters. 

The marginal likelihood, or evidence, for the hyperparameters is obtained by 
integrating over the network weights 


p(Dla, 8) = f pbe law: (5.174) 


This is easily evaluated by making use of the Laplace approximation result (4.135). 
Taking logarithms then gives 


1 W N N 
In p(Dla, 3) ~ —E(wmap) 5 PIA aa Ina4 5 ln 8 5 In(27) (5.175) 


where W is the total number of parameters in w, and the regularized error function 
is defined by 


ooh 


N 
a 
E(wmap) = 7 dtu Y(Xn, WMAP) = th}? T 3 WMAPWMAP: (5.176) 


We see that this takes the same form as the corresponding result (3.86) for the linear 
regression model. 
In the evidence framework, we make point estimates for a and 8 by maximizing 
In p(D|a, 8). Consider first the maximization with respect to a, which can be done 
by analogy with the linear regression case discussed in Section 3.5.2. We first define 
the eigenvalue equation 
GHu; = AGU; (5.177) 
where H is the Hessian matrix comprising the second derivatives of the sum-of- 


squares error function, evaluated at w = wmap. By analogy with (3.92), we obtain 


tes ale = 2 (5.178) 


T 
WMAPWMAP 
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where y represents the effective number of parameters and is defined by 


Ww X, 
= t 5.179 
y Da ( ) 


Note that this result was exact for the linear regression case. For the nonlinear neural 
network, however, it ignores the fact that changes in a will cause changes in the 
Hessian H, which in turn will change the eigenvalues. We have therefore implicitly 
ignored terms involving the derivatives of A; with respect to a. 

Similarly, from (3.95) we see that maximizing the evidence with respect to 3 
gives the re-estimation formula 


N 
; =a > - y enan aT. (5.180) 
n=1 

As with the linear model, we need to alternate between re-estimation of the hyper- 
parameters a and ( and updating of the posterior distribution. The situation with 
a neural network model is more complex, however, due to the multimodality of the 
posterior distribution. As a consequence, the solution for wmap found by maximiz- 
ing the log posterior will depend on the initialization of w. Solutions that differ only 
as a consequence of the interchange and sign reversal symmetries in the hidden units 
are identical so far as predictions are concerned, and it is irrelevant which of the 
equivalent solutions is found. However, there may be inequivalent solutions as well, 
and these will generally yield different values for the optimized hyperparameters. 

In order to compare different models, for example neural networks having differ- 
ent numbers of hidden units, we need to evaluate the model evidence p(D). This can 
be approximated by taking (5.175) and substituting the values of a and 8 obtained 
from the iterative optimization of these hyperparameters. A more careful evaluation 
is obtained by marginalizing over a and (3, again by making a Gaussian approxima- 
tion (MacKay, 1992c; Bishop, 1995a). In either case, it is necessary to evaluate the 
determinant |A | of the Hessian matrix. This can be problematic in practice because 
the determinant, unlike the trace, is sensitive to the small eigenvalues that are often 
difficult to determine accurately. 

The Laplace approximation is based on a local quadratic expansion around a 
mode of the posterior distribution over weights. We have seen in Section 5.1.1 that 
any given mode in a two-layer network is a member of a set of M!2™ equivalent 
modes that differ by interchange and sign-change symmetries, where M is the num- 
ber of hidden units. When comparing networks having different numbers of hid- 
den units, this can be taken into account by multiplying the evidence by a factor of 
Min, 


5.7.3 Bayesian neural networks for classification 


So far, we have used the Laplace approximation to develop a Bayesian treat- 
ment of neural network regression models. We now discuss the modifications to 
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this framework that arise when it is applied to classification. Here we shall con- 
sider a network having a single logistic sigmoid output corresponding to a two-class 
classification problem. The extension to networks with multiclass softmax outputs 
is straightforward. We shall build extensively on the analogous results for linear 
classification models discussed in Section 4.5, and so we encourage the reader to 
familiarize themselves with that material before studying this section. 

The log likelihood function for this model is given by 


Inp(D|w) = X` = 1” {tp lnyn + (1— tn) In(1 — yn) } (5.181) 


nm 


where tn € {0,1} are the target values, and yn = y(Xn,w). Note that there is no 
hyperparameter 8, because the data points are assumed to be correctly labelled. As 
before, the prior is taken to be an isotropic Gaussian of the form (5.162). 

The first stage in applying the Laplace framework to this model is to initialize 
the hyperparameter a, and then to determine the parameter vector w by maximizing 
the log posterior distribution. This is equivalent to minimizing the regularized error 
function a 

E(w) = —Inp(D|w) + aww (5.182) 


and can be achieved using error backpropagation combined with standard optimiza- 
tion algorithms, as discussed in Section 5.3. 

Having found a solution Wap for the weight vector, the next step is to eval- 
uate the Hessian matrix H comprising the second derivatives of the negative log 
likelihood function. This can be done, for instance, using the exact method of Sec- 
tion 5.4.5, or using the outer product approximation given by (5.85). The second 
derivatives of the negative log posterior can again be written in the form (5.166), and 
the Gaussian approximation to the posterior is then given by (5.167). 

To optimize the hyperparameter a, we again maximize the marginal likelihood, 
which is easily shown to take the form 


1 W 
Inp(D\a) ~ —E(wmap) 5 PIA ae Ina + const (5.183) 


where the regularized error function is defined by 


(a7 
E(wuap) =-Soth Inyn + (1 tn) In(1 ~ yn)} + Z WmapWmap (5.184) 


in which yn = y(Xn,Wmap). Maximizing this evidence function with respect to a 
again leads to the re-estimation equation given by (5.178). 

The use of the evidence procedure to determine a is illustrated in Figure 5.22 
for the synthetic two-dimensional data discussed in Appendix A. 

Finally, we need the predictive distribution, which is defined by (5.168). Again, 
this integration is intractable due to the nonlinearity of the network function. The 
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Illustration of the evidence framework 


applied to a synthetic two-class data set. 2 ' " 
The green curve shows the optimal de- 

cision boundary, the black curve shows 2f Oo o® 
the result of fitting a two-layer network o 

with 8 hidden units by maximum likeli- 1f Si 


hood, and the red curve shows the re- 
sult of including a regularizer in which Ot 
a is optimized using the evidence pro- 
cedure, starting from the initial value _, | © 
a = 0. Note that the evidence proce- 
dure greatly reduces the over-fitting of 
the network. 


-2 =1 


simplest approximation is to assume that the posterior distribution is very narrow 
and hence make the approximation 


p(t|x,D) ~ p(t|x, wWmaP). (5.185) 


We can improve on this, however, by taking account of the variance of the posterior 
distribution. In this case, a linear approximation for the network outputs, as was used 
in the case of regression, would be inappropriate due to the logistic sigmoid output- 
unit activation function that constrains the output to lie in the range (0,1). Instead, 
we make a linear approximation for the output unit activation in the form 


a(x,Ww) œ amap (xX) + b'(w — WMAP) (5.186) 


where amap (xX) = a(x, Wap), and the vector b = Va(x, wmap) can be found by 
backpropagation. 

Because we now have a Gaussian approximation for the posterior distribution 
over w, and a model for a that is a linear function of w, we can now appeal to the 
results of Section 4.5.2. The distribution of output unit activation values, induced by 
the distribution over network weights, is given by 


p(a|x, D) = fo (a — amap(x) — b*(x)(w — wmar)) q(w|D)dw (5.187) 


where q(w|D) is the Gaussian approximation to the posterior distribution given by 
(5.167). From Section 4.5.2, we see that this distribution is Gaussian with mean 
amapP = a(x, Wap), and variance 

o2(x) = b"(x)A~*b(x). (5.188) 


a 


Finally, to obtain the predictive distribution, we must marginalize over a using 


p(t = 1|x,D) = f eotalx.D) da. (5.189) 
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Figure 5.23 An illustration of the Laplace approximation for a Bayesian neural network having 8 hidden units 
with ‘tanh’ activation functions and a single logistic-sigmoid output unit. The weight parameters were found using 
scaled conjugate gradients, and the hyperparameter a was optimized using the evidence framework. On the left 
is the result of using the simple approximation (5.185) based on a point estimate wmap of the parameters, 
in which the green curve shows the y = 0.5 decision boundary, and the other contours correspond to output 
probabilities of y = 0.1, 0.3, 0.7, and 0.9. On the right is the corresponding result obtained using (5.190). Note 
that the effect of marginalization is to spread out the contours and to make the predictions less confident, so 
that at each input point x, the posterior probabilities are shifted towards 0.5, while the y = 0.5 contour itself is 


unaffected. 


Exercises 
5.1 


5.2 


The convolution of a Gaussian with a logistic sigmoid is intractable. We therefore 
apply the approximation (4.153) to (5.189) giving 


p(t = 1|x, D) = o (K(o2)b" wap) (5.190) 


where «(-) is defined by (4.154). Recall that both o? and b are functions of x. 
Figure 5.23 shows an example of this framework applied to the synthetic classi- 
fication data set described in Appendix A. 


(xx) Consider a two-layer network function of the form (5.7) in which the hidden- 
unit nonlinear activation functions g(-) are given by logistic sigmoid functions of the 
form 

o(a) = {1 +exp(—a)}"*. (5.191) 


Show that there exists an equivalent network, which computes exactly the same func- 
tion, but with hidden unit activation functions given by tanh(a) where the tanh func- 
tion is defined by (5.59). Hint: first find the relation between c(a) and tanh(a), and 
then show that the parameters of the two networks differ by linear transformations. 


(x) FQ Show that maximizing the likelihood function under the conditional 
distribution (5.16) for a multioutput neural network is equivalent to minimizing the 
sum-of-squares error function (5.11). 


5.3 


5.4 


5.5 


5.6 


5.7 


5.8 


5.9 


5.10 
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(xx) Consider a regression problem involving multiple target variables in which it 
is assumed that the distribution of the targets, conditioned on the input vector x, is a 
Gaussian of the form 

p(t|x, w) = N(tly(x, w), £) (5.192) 


where y(x, w) is the output of a neural network with input vector x and weight 
vector w, and & is the covariance of the assumed Gaussian noise on the targets. 
Given a set of independent observations of x and t, write down the error function 
that must be minimized in order to find the maximum likelihood solution for w, if 
we assume that © is fixed and known. Now assume that & is also to be determined 
from the data, and write down an expression for the maximum likelihood solution 
for X. Note that the optimizations of w and © are now coupled, in contrast to the 
case of independent target variables discussed in Section 5.2. 


(xx) Consider a binary classification problem in which the target values are t € 
{0,1}, with a network output y(x, w) that represents p(t = 1|x), and suppose that 
there is a probability e that the class label on a training data point has been incorrectly 
set. Assuming independent and identically distributed data, write down the error 
function corresponding to the negative log likelihood. Verify that the error function 
(5.21) is obtained when e = 0. Note that this error function makes the model robust 
to incorrectly labelled data, in contrast to the usual error function. 


œ EY Show that maximizing likelihood for a multiclass neural network model 
in which the network outputs have the interpretation y,(x,w) = p(tk = 1|x) is 
equivalent to the minimization of the cross-entropy error function (5.24). 


(x) Y= Show the derivative of the error function (5.21) with respect to the 
activation ax for an output unit having a logistic sigmoid activation function satisfies 
(5.18). 


(x) Show the derivative of the error function (5.24) with respect to the activation a, 
for output units having a softmax activation function satisfies (5.18). 


(x) We saw in (4.88) that the derivative of the logistic sigmoid activation function 
can be expressed in terms of the function value itself. Derive the corresponding result 
for the ‘tanh’ activation function defined by (5.59). 


(x) | www | The error function (5.21) for binary classification problems was de- 
rived for a network having a logistic-sigmoid output activation function, so that 
0 < y(x,w) < 1, and data having target values t € {0,1}. Derive the correspond- 
ing error function if we consider a network having an output —1 < y(x,w) < 1 
and target values t = 1 for class Cı and t = —1 for class C2. What would be the 
appropriate choice of output unit activation function? 


(x) PQQ Consider a Hessian matrix H with eigenvector equation (5.33). By 
setting the vector v in (5.39) equal to each of the eigenvectors u; in turn, show that 
H is positive definite if, and only if, all of its eigenvalues are positive. 
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5.11 


5.12 


5.13 


5.14 


5.15 


5.16 


5.17 


5.18 


5.19 


(« x) www | Consider a quadratic error function defined by (5.32), in which the 
Hessian matrix H has an eigenvalue equation given by (5.33). Show that the con- 
tours of constant error are ellipses whose axes are aligned with the eigenvectors u;, 
with lengths that are inversely proportional to the square root of the corresponding 
eigenvalues \;. 


(*x) www | By considering the local Taylor expansion (5.32) of an error function 
about a stationary point w*, show that the necessary and sufficient condition for the 
stationary point to be a local minimum of the error function is that the Hessian matrix 
H, defined by (5.30) with w = w*, be positive definite. 


(*) Show that as a consequence of the symmetry of the Hessian matrix H, the 
number of independent elements in the quadratic error function (5.28) is given by 
W(W + 3)/2. 


(x) By making a Taylor expansion, verify that the terms that are O(€) cancel on the 
right-hand side of (5.69). 


(xx) In Section 5.3.4, we derived a procedure for evaluating the Jacobian matrix of a 
neural network using a backpropagation procedure. Derive an alternative formalism 
for finding the Jacobian based on forward propagation equations. 


(x) The outer product approximation to the Hessian matrix for a neural network 
using a sum-of-squares error function is given by (5.84). Extend this result to the 
case of multiple outputs. 


(x) Consider a squared loss function of the form 
1 
E=5 f {y(x, w) — t}? p(x, t) dxdt (5.193) 


where y(x, w) is a parametric function such as a neural network. The result (1.89) 
shows that the function y(x, w) that minimizes this error is given by the conditional 
expectation of t given x. Use this result to show that the second derivative of Æ with 
respect to two elements w, and w, of the vector w, is given by 


OE Oy Oy 
nee f TA Ja dx. (5.194) 


Note that, for a finite sample from p(x), we obtain (5.84). 


(x) Consider a two-layer network of the form shown in Figure 5.1 with the addition 
of extra parameters corresponding to skip-layer connections that go directly from 
the inputs to the outputs. By extending the discussion of Section 5.3.2, write down 
the equations for the derivatives of the error function with respect to these additional 
parameters. 


(x) PQQ Derive the expression (5.85) for the outer product approximation to 
the Hessian matrix for a network having a single output with a logistic sigmoid 
output-unit activation function and a cross-entropy error function, corresponding to 
the result (5.84) for the sum-of-squares error function. 


5.20 


5.21 


5.22 


5.23 


5.24 


5.25 
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(x) Derive an expression for the outer product approximation to the Hessian matrix 
for a network having K outputs with a softmax output-unit activation function and 
a cross-entropy error function, corresponding to the result (5.84) for the sum-of- 
squares error function. 


(xxx) Extend the expression (5.86) for the outer product approximation of the Hes- 
sian matrix to the case of K > 1 output units. Hence, derive a recursive expression 
analogous to (5.87) for incrementing the number N of patterns and a similar expres- 
sion for incrementing the number K of outputs. Use these results, together with the 
identity (5.88), to find sequential update expressions analogous to (5.89) for finding 
the inverse of the Hessian by incrementally including both extra patterns and extra 
outputs. 


(xx) Derive the results (5.93), (5.94), and (5.95) for the elements of the Hessian 
matrix of a two-layer feed-forward network by application of the chain rule of cal- 
culus. 


(xx) Extend the results of Section 5.4.5 for the exact Hessian of a two-layer network 
to include skip-layer connections that go directly from inputs to outputs. 


(x) Verify that the network function defined by (5.113) and (5.114) is invariant un- 
der the transformation (5.115) applied to the inputs, provided the weights and biases 
are simultaneously transformed using (5.116) and (5.117). Similarly, show that the 
network outputs can be transformed according (5.118) by applying the transforma- 
tion (5.119) and (5.120) to the second-layer weights and biases. 


œx» QQ Consider a quadratic error function of the form 
1 
E = Eo + 5 (Ww — w*)'H(w — w*) (5.195) 


where w* represents the minimum, and the Hessian matrix H is positive definite and 
constant. Suppose the initial weight vector w is chosen to be at the origin and is 
updated using simple gradient descent 


w =w) _ pVE (5.196) 


where 7 denotes the step number, and p is the learning rate (which is assumed to be 
small). Show that, after 7 steps, the components of the weight vector parallel to the 
eigenvectors of H can be written 


wP = {1 — (1— pnj)"} wt (5.197) 


where w; = wuj, and u; and n; are the eigenvectors and eigenvalues, respectively, 
of H so that 
Hu; = 15 U;- (5.198) 


Show that as T — 00, this gives w‘") — w* as expected, provided |1 — pnj| < 1. 
Now suppose that training is halted after a finite number 7 of steps. Show that the 


288 


5. NEURAL NETWORKS 


5.26 


components of the weight vector parallel to the eigenvectors of the Hessian satisfy 
wi ~ ws when n > (pr)! (5.199) 
lw] <|w%| when nj < (pr). (5.200) 


Compare this result with the discussion in Section 3.5.3 of regularization with simple 
weight decay, and hence show that (pT) ~! is analogous to the regularization param- 
eter \. The above results also show that the effective number of parameters in the 
network, as defined by (3.91), grows as the training progresses. 


(xx) Consider a multilayer perceptron with arbitrary feed-forward topology, which 
is to be trained by minimizing the tangent propagation error function (5.127) in 
which the regularizing function is given by (5.128). Show that the regularization 
term Q can be written as a sum over patterns of terms of the form 


1 
On = 5 2 (Guk)? (5.201) 


where G is a differential operator defined by 
o 
G= a (5.202) 
By acting on the forward propagation equations 


zj = h(a;), ü= $ we (5.203) 


with the operator G, show that Q, can be evaluated by forward propagation using 
the following equations: 


aj = h'(a;) G5, bj = X Uja (5.204) 


where we have defined the new variables 
Qj = Gz;, bj = Gaj. (5.205) 


Now show that the derivatives of Q, with respect to a weight w,., in the network can 
be written in the form 


OD, 
er >, Ak {krZs + kris} (5.206) 
where we have defined 
Yk 
ieee) e = Oike (5.207) 
Oa, 


Write down the backpropagation equations for 6,,, and hence derive a set of back- 
propagation equations for the evaluation of the kr. 
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5.30 
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5.33 


5.34 


5.35 


5.36 


5.37 


5.38 
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œ» FQ Consider the framework for training with transformed data in the 
special case in which the transformation consists simply of the addition of random 
noise x — x + & where € has a Gaussian distribution with zero mean and unit 
covariance. By following an argument analogous to that of Section 5.5.5, show that 
the resulting regularizer reduces to the Tikhonov form (5.135). 


(x) www | Consider a neural network, such as the convolutional network discussed 
in Section 5.5.6, in which multiple weights are constrained to have the same value. 
Discuss how the standard backpropagation algorithm must be modified in order to 
ensure that such constraints are satisfied when evaluating the derivatives of an error 
function with respect to the adjustable parameters in the network. 


(x) FY Verify the result (5.141). 
(x) Verify the result (5.142). 
(x) Verify the result (5.143). 


(xx) Show that the derivatives of the mixing coefficients {7p }, defined by (5.146), 
with respect to the auxiliary parameters {7;} are given by 


F = OjkTj — NjTk. (5.208) 


Hence, by making use of the constraint } > p Tk = 1, derive the result (5.147). 


(x) Write down a pair of equations that express the Cartesian coordinates (21, £2) 
for the robot arm shown in Figure 5.18 in terms of the joint angles 6; and 02 and 
the lengths Lı and Lə of the links. Assume the origin of the coordinate system is 
given by the attachment point of the lower arm. These equations define the ‘forward 
kinematics’ of the robot arm. 


œ FQ Derive the result (5.155) for the derivative of the error function with 
respect to the network output activations controlling the mixing coefficients in the 
mixture density network. 


(x) Derive the result (5.156) for the derivative of the error function with respect 
to the network output activations controlling the component means in the mixture 
density network. 


(x) Derive the result (5.157) for the derivative of the error function with respect to 
the network output activations controlling the component variances in the mixture 
density network. 


(x) Verify the results (5.158) and (5.160) for the conditional mean and variance of 
the mixture density network model. 


(x) Using the general result (2.115), derive the predictive distribution (5.172) for 
the Laplace approximation to the Bayesian neural network model. 
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5.39 


5.40 


5.41 


(x) FQ Make use of the Laplace approximation result (4.135) to show that the 
evidence function for the hyperparameters a and p in the Bayesian neural network 
model can be approximated by (5.175). 


(x) FY Outline the modifications needed to the framework for Bayesian neural 
networks, discussed in Section 5.7.3, to handle multiclass problems using networks 
having softmax output-unit activation functions. 


(xx) By following analogous steps to those given in Section 5.7.1 for regression 
networks, derive the result (5.183) for the marginal likelihood in the case of a net- 
work having a cross-entropy error function and logistic-sigmoid output-unit activa- 
tion function. 


Chapter 5 


Section 2.5.1 


In Chapters 3 and 4, we considered linear parametric models for regression and 
classification in which the form of the mapping y(x, w) from input x to output y 
is governed by a vector w of adaptive parameters. During the learning phase, a 
set of training data is used either to obtain a point estimate of the parameter vector 
or to determine a posterior distribution over this vector. The training data is then 
discarded, and predictions for new inputs are based purely on the learned parameter 
vector w. This approach is also used in nonlinear parametric models such as neural 
networks. 

However, there is a class of pattern recognition techniques, in which the training 
data points, or a subset of them, are kept and used also during the prediction phase. 
For instance, the Parzen probability density model comprised a linear combination 
of ‘kernel’ functions each one centred on one of the training data points. Similarly, 
in Section 2.5.2 we introduced a simple technique for classification called nearest 
neighbours, which involved assigning to each new test vector the same label as the 
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Chapter 7 


Section 12.3 


Section 6.3 


closest example from the training set. These are examples of memory-based methods 
that involve storing the entire training set in order to make predictions for future data 
points. They typically require a metric to be defined that measures the similarity of 
any two vectors in input space, and are generally fast to ‘train’ but slow at making 
predictions for test data points. 

Many linear parametric models can be re-cast into an equivalent ‘dual represen- 
tation’ in which the predictions are also based on linear combinations of a kernel 
Junction evaluated at the training data points. As we shall see, for models which are 
based on a fixed nonlinear feature space mapping p(x), the kernel function is given 
by the relation 

k(x, x’) = (x) *(x’). (6.1) 


From this definition, we see that the kernel is a symmetric function of its arguments 
so that k(x, x’) = k(x’, x). The kernel concept was introduced into the field of pat- 
tern recognition by Aizerman et al. (1964) in the context of the method of potential 
functions, so-called because of an analogy with electrostatics. Although neglected 
for many years, it was re-introduced into machine learning in the context of large- 
margin classifiers by Boser et al. (1992) giving rise to the technique of support 
vector machines. Since then, there has been considerable interest in this topic, both 
in terms of theory and applications. One of the most significant developments has 
been the extension of kernels to handle symbolic objects, thereby greatly expanding 
the range of problems that can be addressed. 

The simplest example of a kernel function is obtained by considering the identity 
mapping for the feature space in (6.1) so that @(x) = x, in which case k(x, x’) = 
xx’. We shall refer to this as the linear kernel. 

The concept of a kernel formulated as an inner product in a feature space allows 
us to build interesting extensions of many well-known algorithms by making use of 
the kernel trick, also known as kernel substitution. The general idea is that, if we have 
an algorithm formulated in such a way that the input vector x enters only in the form 
of scalar products, then we can replace that scalar product with some other choice of 
kernel. For instance, the technique of kernel substitution can be applied to principal 
component analysis in order to develop a nonlinear variant of PCA (Scholkopf et al., 
1998). Other examples of kernel substitution include nearest-neighbour classifiers 
and the kernel Fisher discriminant (Mika et al., 1999; Roth and Steinhage, 2000; 
Baudat and Anouar, 2000). 

There are numerous forms of kernel functions in common use, and we shall en- 
counter several examples in this chapter. Many have the property of being a function 
only of the difference between the arguments, so that k(x,x’) = k(x — x’), which 
are known as stationary kernels because they are invariant to translations in input 
space. A further specialization involves homogeneous kernels, also known as ra- 
dial basis functions, which depend only on the magnitude of the distance (typically 
Euclidean) between the arguments so that k(x, x’) = k(||x — x’||). 

For recent textbooks on kernel methods, see Scholkopf and Smola (2002), Her- 
brich (2002), and Shawe-Taylor and Cristianini (2004). 


6.1. Dual Representations 293 


6.1. Dual Representations 


Many linear models for regression and classification can be reformulated in terms of 
a dual representation in which the kernel function arises naturally. This concept will 
play an important role when we consider support vector machines in the next chapter. 
Here we consider a linear regression model whose parameters are determined by 
minimizing a regularized sum-of-squares error function given by 


N 
J(w) = i > {w" (Xn) — tae + swhw (6.2) 
n=1 


where À > 0. If we set the gradient of J(w) with respect to w equal to zero, we see 
that the solution for w takes the form of a linear combination of the vectors @(x,), 
with coefficients that are functions of w, of the form 


N N 
1 

vo Ly {w" (Xn) = tn} ax) = > An@(Xn) = PTa (6.3) 

n=1 n=1 
where ® is the design matrix, whose n‘” row is given by #(x,,)?. Here the vector 

a = (a1, ..., an)", and we have defined 
1 

m=- {w" (xn) —tn}. (6.4) 


Instead of working with the parameter vector w, we can now reformulate the least- 
squares algorithm in terms of the parameter vector a, giving rise to a dual represen- 
tation. If we substitute w = "a into J(w), we obtain 


1 1 A 
J(a) = sa So ega —a @6't+ st + za Pla (6.5) 
where t = (t1,..., tn). We now define the Gram matrix K = ®&", which is an 
N x N symmetric matrix with elements 
Kam — (Xn) (Xm) = k(Xn, Xm) (6.6) 


where we have introduced the kernel function k(x, x’) defined by (6.1). In terms of 
the Gram matrix, the sum-of-squares error function can be written as 


J(a) = 5a"KKa a’ Kt4 tt na" Ka. (6.7) 


Setting the gradient of J(a) with respect to a to zero, we obtain the following solu- 
tion 
a=(K+AIy) t. (6.8) 
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6.2. 


If we substitute this back into the linear regression model, we obtain the following 
prediction for a new input x 


y(x) = w' d(x) = a ®(x) = k(x)? (K+ Aly) `t (6.9) 


where we have defined the vector k(x) with elements kn(x) = k(xn,x). Thus we 
see that the dual formulation allows the solution to the least-squares problem to be 
expressed entirely in terms of the kernel function k(x, x’). This is known as a dual 
formulation because, by noting that the solution for a can be expressed as a linear 
combination of the elements of @(x), we recover the original formulation in terms of 
the parameter vector w. Note that the prediction at x is given by a linear combination 
of the target values from the training set. In fact, we have already obtained this result, 
using a slightly different notation, in Section 3.3.3. 

In the dual formulation, we determine the parameter vector a by inverting an 
N x N matrix, whereas in the original parameter space formulation we had to invert 
an M x M matrix in order to determine w. Because XN is typically much larger 
than M, the dual formulation does not seem to be particularly useful. However, the 
advantage of the dual formulation, as we shall see, is that it is expressed entirely in 
terms of the kernel function k(x, x’). We can therefore work directly in terms of 
kernels and avoid the explicit introduction of the feature vector @(x), which allows 
us implicitly to use feature spaces of high, even infinite, dimensionality. 

The existence of a dual representation based on the Gram matrix is a property of 
many linear models, including the perceptron. In Section 6.4, we will develop a dual- 
ity between probabilistic linear models for regression and the technique of Gaussian 
processes. Duality will also play an important role when we discuss support vector 
machines in Chapter 7. 


Constructing Kernels 


In order to exploit kernel substitution, we need to be able to construct valid kernel 
functions. One approach is to choose a feature space mapping (x) and then use 
this to find the corresponding kernel, as is illustrated in Figure 6.1. Here the kernel 
function is defined for a one-dimensional input space by 


M 
k(x, 2’) = 6(2)"(2’) = > tagl’) (6.10) 


where @; (a) are the basis functions. 

An alternative approach is to construct kernel functions directly. In this case, 
we must ensure that the function we choose is a valid kernel, in other words that it 
corresponds to a scalar product in some (perhaps infinite dimensional) feature space. 
As a simple example, consider a kernel function given by 


k(x,z) = (xTz)*. (6.11) 
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Figure 6.1 Illustration of the construction of kernel functions starting from a corresponding set of basis func- 


tions. In each column the lower plot shows the kernel function k(x, x’) defined by (6.10) plotted as a function of 
x for x’ = 0, while the upper plot shows the corresponding basis functions given by polynomials (left column), 
‘Gaussians’ (centre column), and logistic sigmoids (right column). 


Appendix C 


If we take the particular case of a two-dimensional input space x = (21,22) we 
can expand out the terms and thereby identify the corresponding nonlinear feature 


mapping 


k(x, z) (x™z)° = (£121 + T222) 


= xz? + 29121022 + riza 
(x?, V 22429, £2)(2z?, / 22129, 22)7 
p(x)" P(z). (6.12) 


We see that the feature mapping takes the form ¢(x) = (x2, /2x1%2, 23)? and 
therefore comprises all possible second order terms, with a specific weighting be- 
tween them. 

More generally, however, we need a simple way to test whether a function con- 
stitutes a valid kernel without having to construct the function @(x) explicitly. A 
necessary and sufficient condition for a function k(x, x’) to be a valid kernel (Shawe- 
Taylor and Cristianini, 2004) is that the Gram matrix K, whose elements are given by 
k(Xn, Xm), should be positive semidefinite for all possible choices of the set {xn }. 
Note that a positive semidefinite matrix is not the same thing as a matrix whose 
elements are nonnegative. 

One powerful technique for constructing new kernels is to build them out of 
simpler kernels as building blocks. This can be done using the following properties: 
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Given valid kernels k(x, x’) and k2(x, x’), the following new kernels will also 
be valid: 


kx} ckı(x, x’) (6.13) 
k(x,x) = f(x)kı(x, x) f(x’) (6.14) 
hoes). = q(kı(x,x’)) (6.15) 
k(x,x’) = exp(kı(x,x’)) (6.16) 
k(x,x’) = ki(x,x’) + ko(x,x’) (6.17) 
koe) = kil x eax) (6.18) 
k(x x’) = ka (Olx), d(x’) (6.19) 
k(x,x’) = x"Ax' (6.20) 
ki) = kal(Xa, X4) + hae, (6.21) 
k(x, x’) = ka(Xa, en X$) (6.22) 


where c > 0 is a constant, f(-) is any function, q(-) is a polynomial with nonneg- 
ative coefficients, @(x) is a function from x to R™, k3(-,-) is a valid kernel in 
R“, Aisa symmetric positive semidefinite matrix, Xa and x, are variables (not 
necessarily disjoint) with x = (Xa, Xb), and ka and ky are valid kernel functions 
over their respective spaces. 


— 


Equipped with these properties, we can now embark on the construction of more 
complex kernels appropriate to specific applications. We require that the kernel 
k(x, x’) be symmetric and positive semidefinite and that it expresses the appropriate 
form of similarity between x and x’ according to the intended application. Here we 
consider a few common examples of kernel functions. For a more extensive discus- 
sion of ‘kernel engineering’, see Shawe-Taylor and Cristianini (2004). 

We saw that the simple polynomial kernel k(x,x’) = (xTx' j contains only 
terms of degree two. If we consider the slightly generalized kernel k(x, x’) = 


(x + c) ? with c > 0, then the corresponding feature mapping @(x) contains con- 


stant and linear terms as well as terms of order two. Similarly, k(x, x’) = (xTx ) = 
contains all monomials of order M. For instance, if x and x’ are two images, then 
the kernel represents a particular weighted sum of all possible products of M pixels 
in the first image with M pixels in the second image. This can similarly be gener- 
alized to include all terms up to degree M by considering k(x, x’) = (xT + c) = 
with c > 0. Using the results (6.17) and (6.18) for combining kernels we see that 
these will all be valid kernel functions. 
Another commonly used kernel takes the form 


k(x,x’) = exp (—||x — x'||?/207) (6.23) 


and is often called a ‘Gaussian’ kernel. Note, however, that in this context it is 
not interpreted as a probability density, and hence the normalization coefficient is 


Exercise 6.11 


Exercise 6.12 
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omitted. We can see that this is a valid kernel by expanding the square 
|x — x’ ||? = xTx + (x’)Tx’ — 2x7x’ (6.24) 
to give 
k(x, x’) = exp (—x"x/207) exp (x"x’/o7) exp (—(x’)™x’ /207) (6.25) 


and then making use of (6.14) and (6.16), together with the validity of the linear 
kernel k(x, x’) = x?x’. Note that the feature vector that corresponds to the Gaussian 
kernel has infinite dimensionality. 

The Gaussian kernel is not restricted to the use of Euclidean distance. If we use 
kernel substitution in (6.24) to replace xx’ with a nonlinear kernel «(x,x’), we 
obtain 

k(x, x’) = exp f- (k(x, x) + K(x’, x’) — 2n) } ; (6.26) 

An important contribution to arise from the kernel viewpoint has been the exten- 
sion to inputs that are symbolic, rather than simply vectors of real numbers. Kernel 
functions can be defined over objects as diverse as graphs, sets, strings, and text doc- 
uments. Consider, for instance, a fixed set and define a nonvectorial space consisting 
of all possible subsets of this set. If A, and A» are two such subsets then one simple 
choice of kernel would be 


k(A1, Ap) = 2!41749l (6.27) 


where A; N A> denotes the intersection of sets A; and A», and |A| denotes the 
number of subsets in A. This is a valid kernel function because it can be shown to 
correspond to an inner product in a feature space. 

One powerful approach to the construction of kernels starts from a probabilistic 
generative model (Haussler, 1999), which allows us to apply generative models in a 
discriminative setting. Generative models can deal naturally with missing data and 
in the case of hidden Markov models can handle sequences of varying length. By 
contrast, discriminative models generally give better performance on discriminative 
tasks than generative models. It is therefore of some interest to combine these two 
approaches (Lasserre et al., 2006). One way to combine them is to use a generative 
model to define a kernel, and then use this kernel in a discriminative approach. 

Given a generative model p(x) we can define a kernel by 


k(x,x’) = p(x)p(x’). (6.28) 


This is clearly a valid kernel function because we can interpret it as an inner product 
in the one-dimensional feature space defined by the mapping p(x). It says that two 
inputs x and x’ are similar if they both have high probabilities. We can use (6.13) and 
(6.17) to extend this class of kernels by considering sums over products of different 
probability distributions, with positive weighting coefficients p(i), of the form 


fags y= do Plxlie(x lp: (6.29) 
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This is equivalent, up to an overall multiplicative constant, to a mixture distribution 
in which the components factorize, with the index 7 playing the role of a ‘latent’ 
variable. Two inputs x and x’ will give a large value for the kernel function, and 
hence appear similar, if they have significant probability under a range of different 
components. Taking the limit of an infinite sum, we can also consider kernels of the 
form 


k(x, x’) = [ vizo) dz (6.30) 


where z is a continuous latent variable. 

Now suppose that our data consists of ordered sequences of length L so that 
an observation is given by X = {x1,..., XL}. A popular generative model for 
sequences is the hidden Markov model, which expresses the distribution p(X) as a 
marginalization over a corresponding sequence of hidden states Z = {z1,..., Zz}. 
We can use this approach to define a kernel function measuring the similarity of two 
sequences X and X’ by extending the mixture representation (6.29) to give 


k(X,X’) = X p(X|Z)p(X'|Z)p(Z) (6.31) 
Z 

so that both observed sequences are generated by the same hidden sequence Z. This 
model can easily be extended to allow sequences of differing length to be compared. 
An alternative technique for using generative models to define kernel functions 
is known as the Fisher kernel (Jaakkola and Haussler, 1999). Consider a parametric 
generative model p(x|@) where @ denotes the vector of parameters. The goal is to 
find a kernel that measures the similarity of two input vectors x and x’ induced by the 
generative model. Jaakkola and Haussler (1999) consider the gradient with respect 
to 8, which defines a vector in a ‘feature’ space having the same dimensionality as 

0. In particular, they consider the Fisher score 


g(0,x) = Vo ln p(x|8) (6.32) 
from which the Fisher kernel is defined by 
k(x, x’) = g(0,x)'F~'g(0,x’). (6.33) 
Here F is the Fisher information matrix, given by 
F = E, |g(9,x)g(9,x)*| (6.34) 


where the expectation is with respect to x under the distribution p(x|@). This can 
be motivated from the perspective of information geometry (Amari, 1998), which 
considers the differential geometry of the space of model parameters. Here we sim- 
ply note that the presence of the Fisher information matrix causes this kernel to be 
invariant under a nonlinear re-parameterization of the density model 0 — (0). 

In practice, it is often infeasible to evaluate the Fisher information matrix. One 
approach is simply to replace the expectation in the definition of the Fisher informa- 
tion with the sample average, giving 


N 
1 
Pa, X g(0,xn)g(0, xn)". (6.35) 


n=l 
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This is the covariance matrix of the Fisher scores, and so the Fisher kernel corre- 
sponds to a whitening of these scores. More simply, we can just omit the Fisher 
information matrix altogether and use the noninvariant kernel 


k(x,x’) = g(0,x)'g(0,x’). (6.36) 


An application of Fisher kernels to document retrieval is given by Hofmann (2000). 
A final example of a kernel function is the sigmoidal kernel given by 


k(x, x’) = tanh (ax™x’ + b) (6.37) 


whose Gram matrix in general is not positive semidefinite. This form of kernel 
has, however, been used in practice (Vapnik, 1995), possibly because it gives kernel 
expansions such as the support vector machine a superficial resemblance to neural 
network models. As we shall see, in the limit of an infinite number of basis functions, 
a Bayesian neural network with an appropriate prior reduces to a Gaussian process, 
thereby providing a deeper link between neural networks and kernel methods. 


Radial Basis Function Networks 


In Chapter 3, we discussed regression models based on linear combinations of fixed 
basis functions, although we did not discuss in detail what form those basis functions 
might take. One choice that has been widely used is that of radial basis functions, 
which have the property that each basis function depends only on the radial distance 
(typically Euclidean) from a centre p}, so that ¢;(x) = h(||x — p,||). 

Historically, radial basis functions were introduced for the purpose of exact func- 
tion interpolation (Powell, 1987). Given a set of input vectors {x,,...,x.} along 
with corresponding target values {t;,...,¢,7}, the goal is to find a smooth function 
f(x) that fits every target value exactly, so that f(x») = tn for n = 1,..., N. This 
is achieved by expressing f(x) as a linear combination of radial basis functions, one 
centred on every data point 


N 
f(x) = X wah((|x — Xnll). (6.38) 


The values of the coefficients {w,,} are found by least squares, and because there 
are the same number of coefficients as there are constraints, the result is a function 
that fits every target value exactly. In pattern recognition applications, however, the 
target values are generally noisy, and exact interpolation is undesirable because this 
corresponds to an over-fitted solution. 

Expansions in radial basis functions also arise from regularization theory (Pog- 
gio and Girosi, 1990; Bishop, 1995a). For a sum-of-squares error function with a 
regularizer defined in terms of a differential operator, the optimal solution is given 
by an expansion in the Green’s functions of the operator (which are analogous to the 
eigenvectors of a discrete matrix), again with one basis function centred on each data 
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point. If the differential operator is isotropic then the Green’s functions depend only 
on the radial distance from the corresponding data point. Due to the presence of the 
regularizer, the solution no longer interpolates the training data exactly. 

Another motivation for radial basis functions comes from a consideration of 
the interpolation problem when the input (rather than the target) variables are noisy 
(Webb, 1994; Bishop, 1995a). If the noise on the input variable x is described 
by a variable € having a distribution v(€), then the sum-of-squares error function 
becomes 


1 N 
E=, > f {y(Xn + E) — tn} v(é) dé. (6.39) 


Using the calculus of variations, we can optimize with respect to the function f(x) 
to give 


N 
YKn = 5 t,h(x — Xn) (6.40) 
n=1 
where the basis functions are given by 
h(x — xn) = U(X = Xn) (6.41) 


N 
5 V(X — Xn) 


We see that there is one basis function centred on every data point. This is known as 
the Nadaraya-Watson model and will be derived again from a different perspective 
in Section 6.3.1. If the noise distribution v(€) is isotropic, so that it is a function 
only of ||£||, then the basis functions will be radial. 

Note that the basis functions (6.41) are normalized, so that }> h(x — Xn) = 1 
for any value of x. The effect of such normalization is shown in Figure 6.2. Normal- 
ization is sometimes used in practice as it avoids having regions of input space where 
all of the basis functions take small values, which would necessarily lead to predic- 
tions in such regions that are either small or controlled purely by the bias parameter. 


Another situation in which expansions in normalized radial basis functions arise 
is in the application of kernel density estimation to the problem of regression, as we 
shall discuss in Section 6.3.1. 

Because there is one basis function associated with every data point, the corre- 
sponding model can be computationally costly to evaluate when making predictions 
for new data points. Models have therefore been proposed (Broomhead and Lowe, 
1988; Moody and Darken, 1989; Poggio and Girosi, 1990), which retain the expan- 
sion in radial basis functions but where the number M of basis functions is smaller 
than the number JN of data points. Typically, the number of basis functions, and the 
locations 1, of their centres, are determined based on the input data {x,, } alone. The 
basis functions are then kept fixed and the coefficients {w;} are determined by least 
squares by solving the usual set of linear equations, as discussed in Section 3.1.1. 
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Figure 6.2 Plot of a set of Gaussian basis functions on the left, together with the corresponding normalized 
basis functions on the right. 
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One of the simplest ways of choosing basis function centres is to use a randomly 
chosen subset of the data points. A more systematic approach is called orthogonal 
least squares (Chen et al., 1991). This is a sequential selection process in which at 
each step the next data point to be chosen as a basis function centre corresponds to 
the one that gives the greatest reduction in the sum-of-squares error. Values for the 
expansion coefficients are determined as part of the algorithm. Clustering algorithms 
such as -means have also been used, which give a set of basis function centres that 
no longer coincide with training data points. 


6.3.1 Nadaraya-Watson model 


In Section 3.3.3, we saw that the prediction of a linear regression model for a 
new input x takes the form of a linear combination of the training set target values 
with coefficients given by the ‘equivalent kernel’ (3.62) where the equivalent kernel 
satisfies the summation constraint (3.64). 

We can motivate the kernel regression model (3.61) from a different perspective, 
starting with kernel density estimation. Suppose we have a training set {x,,,t,,} and 
we use a Parzen density estimator to model the joint distribution p(x, t), so that 


N 
p(x, t) = : 5 fX = st = tn) (6.42) 


where f (x,t) is the component density function, and there is one such component 
centred on each data point. We now find an expression for the regression function 
y(x), corresponding to the conditional average of the target variable conditioned on 
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the input variable, which is given by 


y(x) = Eft 


| p(x, t) dt 


S tænt- n)a 


= : (6.43) 


I | He- kmt- tm) dt 


We now assume for simplicity that the component density functions have zero mean 
so that 


/ f(x, t)tdt =0 (6.44) 


for all values of x. Using a simple change of variable, we then obtain 


S 9X = Xn)tn 


y(x) = 
Y= 9x — Xm) 
= `> k(X, Xain (6.45) 
where n,m = 1,..., N and the kernel function k(x, Xn) is given by 
E T cose (6.46) 
D9 xm) 
and we have defined PA 
g(x) = / FOG t) dt. (6.47) 


The result (6.45) is known as the Nadaraya-Watson model, or kernel regression 
(Nadaraya, 1964; Watson, 1964). For a localized kernel function, it has the prop- 
erty of giving more weight to the data points x,, that are close to x. Note that the 
kernel (6.46) satisfies the summation constraint 


N 
> k(x,X,) = 1. 
n=1 


Figure 6.3 Illustration of the Nadaraya-Watson kernel 
regression model using isotropic Gaussian kernels, for the 
sinusoidal data set. The original sine function is shown 
by the green curve, the data points are shown in blue, 
and each is the centre of an isotropic Gaussian kernel. 
The resulting regression function, given by the condi- 
tional mean, is shown by the red line, along with the two- 
standard-deviation region for the conditional distribution 
p(t|z) shown by the red shading. The blue ellipse around 
each data point shows one standard deviation contour for 
the corresponding kernel. These appear noncircular due 
to the different scales on the horizontal and vertical axes. 
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In fact, this model defines not only a conditional expectation but also a full 
conditional distribution given by 


= (6.48) 


from which other expectations can be evaluated. 

As an illustration we consider the case of a single input variable x in which 
f(a, t) is given by a zero-mean isotropic Gaussian over the variable z = (x,t) with 
variance a”. The corresponding conditional distribution (6.48) is given by a Gaus- 
sian mixture, and is shown, together with the conditional mean, for the sinusoidal 
synthetic data set in Figure 6.3. 

An obvious extension of this model is to allow for more flexible forms of Gaus- 
sian components, for instance having different variance parameters for the input and 
target variables. More generally, we could model the joint distribution p(t, x) using 
a Gaussian mixture model, trained using techniques discussed in Chapter 9 (Ghahra- 
mani and Jordan, 1994), and then find the corresponding conditional distribution 
p(t|x). In this latter case we no longer have a representation in terms of kernel func- 
tions evaluated at the training set data points. However, the number of components 
in the mixture model can be smaller than the number of training set points, resulting 
in a model that is faster to evaluate for test data points. We have thereby accepted an 
increased computational cost during the training phase in order to have a model that 
is faster at making predictions. 


Gaussian Processes 


In Section 6.1, we introduced kernels by applying the concept of duality to a non- 
probabilistic model for regression. Here we extend the role of kernels to probabilis- 
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tic discriminative models, leading to the framework of Gaussian processes. We shall 
thereby see how kernels arise naturally in a Bayesian setting. 

In Chapter 3, we considered linear regression models of the form y(x,w) = 
w(x) in which w is a vector of parameters and #(x) is a vector of fixed nonlinear 
basis functions that depend on the input vector x. We showed that a prior distribution 
over w induced a corresponding prior distribution over functions y(x, w). Given a 
training data set, we then evaluated the posterior distribution over w and thereby 
obtained the corresponding posterior distribution over regression functions, which 
in turn (with the addition of noise) implies a predictive distribution p(t|x) for new 
input vectors x. 

In the Gaussian process viewpoint, we dispense with the parametric model and 
instead define a prior probability distribution over functions directly. At first sight, it 
might seem difficult to work with a distribution over the uncountably infinite space of 
functions. However, as we shall see, for a finite training set we only need to consider 
the values of the function at the discrete set of input values x,, corresponding to the 
training set and test set data points, and so in practice we can work in a finite space. 

Models equivalent to Gaussian processes have been widely studied in many dif- 
ferent fields. For instance, in the geostatistics literature Gaussian process regression 
is known as kriging (Cressie, 1993). Similarly, ARMA (autoregressive moving aver- 
age) models, Kalman filters, and radial basis function networks can all be viewed as 
forms of Gaussian process models. Reviews of Gaussian processes from a machine 
learning perspective can be found in MacKay (1998), Williams (1999), and MacKay 
(2003), and a comparison of Gaussian process models with alternative approaches is 
given in Rasmussen (1996). See also Rasmussen and Williams (2006) for a recent 
textbook on Gaussian processes. 


6.4.1 Linear regression revisited 


In order to motivate the Gaussian process viewpoint, let us return to the linear 
regression example and re-derive the predictive distribution by working in terms 
of distributions over functions y(x,w). This will provide a specific example of a 
Gaussian process. 

Consider a model defined in terms of a linear combination of M fixed basis 
functions given by the elements of the vector @(x) so that 


y(x) = w' f(x) (6.49) 


where x is the input vector and w is the 1/-dimensional weight vector. Now consider 
a prior distribution over w given by an isotropic Gaussian of the form 


p(w) = N(w|0,a7'T) (6.50) 


governed by the hyperparameter a, which represents the precision (inverse variance) 
of the distribution. For any given value of w, the definition (6.49) defines a partic- 
ular function of x. The probability distribution over w defined by (6.50) therefore 
induces a probability distribution over functions y(x). In practice, we wish to eval- 
uate this function at specific values of x, for example at the training data points 
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X1,...,Xy. We are therefore interested in the joint distribution of the function val- 
ues y(X1),...,y(x), which we denote by the vector y with elements yn = y(Xn) 
for n = 1,..., N. From (6.49), this vector is given by 

y= ®w (6.51) 


where ® is the design matrix with elements Png = ox(Xn). We can find the proba- 
bility distribution of y as follows. First of all we note that y is a linear combination of 
Gaussian distributed variables given by the elements of w and hence is itself Gaus- 
sian. We therefore need only to find its mean and covariance, which are given from 
(6.50) by 


Ely] = SE[w]=0 (6.52) 


ik 
covly] = E [yy"] = ®E[ww'] $7 =—#6'=K (6.53) 
- a 
where K is the Gram matrix with elements 
1 
Knm = k(n, Xm) = —G(Kn)! (Xm) (6.54) 


and k(x, x’) is the kernel function. 

This model provides us with a particular example of a Gaussian process. In gen- 
eral, a Gaussian process is defined as a probability distribution over functions y(x) 
such that the set of values of y(x) evaluated at an arbitrary set of points x,,...,xy 
jointly have a Gaussian distribution. In cases where the input vector x is two di- 
mensional, this may also be known as a Gaussian random field. More generally, a 
stochastic process y(x) is specified by giving the joint probability distribution for 
any finite set of values y(x,),...,y(xy) in a consistent manner. 

A key point about Gaussian stochastic processes is that the joint distribution 
over N variables y,..., yn is specified completely by the second-order statistics, 
namely the mean and the covariance. In most applications, we will not have any 
prior knowledge about the mean of y(x) and so by symmetry we take it to be zero. 
This is equivalent to choosing the mean of the prior over weight values p(w|a) to 
be zero in the basis function viewpoint. The specification of the Gaussian process is 
then completed by giving the covariance of y(x) evaluated at any two values of x, 
which is given by the kernel function 


E [y(xn)y(Xm)] = k(Xn, Xm). (6.55) 


For the specific case of a Gaussian process defined by the linear regression model 
(6.49) with a weight prior (6.50), the kernel function is given by (6.54). 

We can also define the kernel function directly, rather than indirectly through a 
choice of basis function. Figure 6.4 shows samples of functions drawn from Gaus- 
sian processes for two different choices of kernel function. The first of these is a 
‘Gaussian’ kernel of the form (6.23), and the second is the exponential kernel given 
by 


k(x, x") = exp (—0 |x — x'|) (6.56) 
which corresponds to the Ornstein-Uhlenbeck process originally introduced by Uh- 
lenbeck and Ornstein (1930) to describe Brownian motion. 
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Figure 6.4 Samples from Gaus- 
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6.4.2 Gaussian processes for regression 


In order to apply Gaussian process models to the problem of regression, we need 
to take account of the noise on the observed target values, which are given by 


tn = Yn + En (6.57) 


where yn = y(Xn), and €n is a random noise variable whose value is chosen inde- 
pendently for each observation n. Here we shall consider noise processes that have 
a Gaussian distribution, so that 


P(tn|Yn) = N (tnlYn, Bg") (6.58) 


where (3 is a hyperparameter representing the precision of the noise. Because the 
noise is independent for each data point, the joint distribution of the target values 


t = (t1,...,t~)7 conditioned on the values of y = (y1, ..., yn)" is given by an 
isotropic Gaussian of the form 
p(tly) = N (tly, 6 "Iy) (6.59) 


where Iy denotes the N x N unit matrix. From the definition of a Gaussian process, 
the marginal distribution p(y) is given by a Gaussian whose mean is zero and whose 
covariance is defined by a Gram matrix K so that 


p(y) = N (y|0, K). (6.60) 


The kernel function that determines K is typically chosen to express the property 
that, for points x, and Xm that are similar, the corresponding values y(x,,) and 
Yy(Xm) will be more strongly correlated than for dissimilar points. Here the notion 
of similarity will depend on the application. 

In order to find the marginal distribution p(t), conditioned on the input values 
X1,.-..,Xw, we need to integrate over y. This can be done by making use of the 
results from Section 2.3.3 for the linear-Gaussian model. Using (2.115), we see that 
the marginal distribution of t is given by 


v(t) = I p(tly)p(y) dy = N (tlo, C) (6.61) 
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where the covariance matrix C has elements 
C(Xn: Xm) = BR Xm) + a (6.62) 


This result reflects the fact that the two Gaussian sources of randomness, namely 
that associated with y(x) and that associated with €, are independent and so their 
covariances simply add. 

One widely used kernel function for Gaussian process regression is given by the 
exponential of a quadratic form, with the addition of constant and linear terms to 
give 


0 
k(Xn, Xm) = 99 exp {— Fhe — xnl? + 62 + 0X1 Xm. (6.63) 


Note that the term involving 93 corresponds to a parametric model that is a linear 
function of the input variables. Samples from this prior are plotted for various values 


of the parameters 090, . . . , 03 in Figure 6.5, and Figure 6.6 shows a set of points sam- 
pled from the joint distribution (6.60) along with the corresponding values defined 
by (6.61). 


So far, we have used the Gaussian process viewpoint to build a model of the 
joint distribution over sets of data points. Our goal in regression, however, is to 
make predictions of the target variables for new inputs, given a set of training data. 
Let us suppose that ty = (t1, .. . , ty)", corresponding to input values x,,...,xy, 
comprise the observed training set, and our goal is to predict the target variable t y+, 
for a new input vector Xy+1. This requires that we evaluate the predictive distri- 
bution p(tv+1|tw). Note that this distribution is conditioned also on the variables 
X1,.-.,Xw and xy+1. However, to keep the notation simple we will not show these 
conditioning variables explicitly. 

To find the conditional distribution p(t) +,|t), we begin by writing down the 
joint distribution p(ty.,), where tyy,, denotes the vector (t;,...,t,t~41)'. We 
then apply the results from Section 2.3.1 to obtain the required conditional distribu- 
tion, as illustrated in Figure 6.7. 

From (6.61), the joint distribution over t1,...,tv41 will be given by 


pP(twa1) = N (tw 41/0, Crv41) (6.64) 


where Cy4+1 is an (N + 1) x (N + 1) covariance matrix with elements given by 
(6.62). Because this joint distribution is Gaussian, we can apply the results from 
Section 2.3.1 to find the conditional Gaussian distribution. To do this, we partition 
the covariance matrix as follows 


C k 
cxn = ( Ke ) (6.65) 


C 


where C y is the N x N covariance matrix with elements given by (6.62) for n, m = 
1,..., N, the vector k has elements k(xn,xXy41) for n = 1,...,N, and the scalar 
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Figure 6.5 Samples from a Gaussian process prior defined by the covariance function (6.63). The title above 
each plot denotes (80, 41, 92, 03). 


c= k(xn41,XNn41) + 6—1. Using the results (2.81) and (2.82), we see that the con- 
ditional distribution p(t +1|t) is a Gaussian distribution with mean and covariance 
given by 

ky! (6.66) 
e—k"Cy’k. (6.67) 


m(X N41) 


o? (xn+1) 


These are the key results that define Gaussian process regression. Because the vector 
k is a function of the test point input value x y1, we see that the predictive distribu- 
tion is a Gaussian whose mean and variance both depend on xy+1. An example of 
Gaussian process regression is shown in Figure 6.8. 

The only restriction on the kernel function is that the covariance matrix given by 
(6.62) must be positive definite. If \; is an eigenvalue of K, then the corresponding 
eigenvalue of C will be \; + 87+. It is therefore sufficient that the kernel matrix 
k(Xn, Xm) be positive semidefinite for any pair of points x,, and Xm, so that A; > 0, 
because any eigenvalue A; that is zero will still give rise to a positive eigenvalue 
for C because 8 > 0. This is the same restriction on the kernel function discussed 
earlier, and so we can again exploit all of the techniques in Section 6.2 to construct 
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Illustration of the sampling of data 
points {tn } from a Gaussian process. 
The blue curve shows a sample func- 
tion from the Gaussian process prior 
over functions, and the red points 
show the values of yn obtained by t 
evaluating the function at a set of in- 
put values {zn}. The correspond- 
ing values of {tn}, shown in green, 
are obtained by adding independent | 
Gaussian noise to each of the {yn}. 
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suitable kernels. 
Note that the mean (6.66) of the predictive distribution can be written, as a func- 
tion of xy +1, in the form 


N 
m(xn41) = D ank(%n,Xw41) (6.68) 


where an is the nt? component of Cyt Thus, if the kernel function k(x,, Xm) 
depends only on the distance ||x,, — Xm ||, then we obtain an expansion in radial 
basis functions. 

The results (6.66) and (6.67) define the predictive distribution for Gaussian pro- 
cess regression with an arbitrary kernel function k(Xn, Xm). In the particular case in 
which the kernel function k(x, x’) is defined in terms of a finite set of basis functions, 
we can derive the results obtained previously in Section 3.3.2 for linear regression 
starting from the Gaussian process viewpoint. 

For such models, we can therefore obtain the predictive distribution either by 
taking a parameter space viewpoint and using the linear regression result or by taking 
a function space viewpoint and using the Gaussian process result. 

The central computational operation in using Gaussian processes will involve 
the inversion of a matrix of size N x N, for which standard methods require O( N°) 
computations. By contrast, in the basis function model we have to invert a matrix 
Sy of size M x M, which has O( M°) computational complexity. Note that for 
both viewpoints, the matrix inversion must be performed once for the given training 
set. For each new test point, both methods require a vector-matrix multiply, which 
has cost O(N?) in the Gaussian process case and O( M?) for the linear basis func- 
tion model. If the number M of basis functions is smaller than the number N of 
data points, it will be computationally more efficient to work in the basis function 
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Figure 6.8 


Illustration of the mechanism of 
Gaussian process regression for 
the case of one training point and 
one test point, in which the red el- 
lipses show contours of the joint dis- 
tribution p(ti,t2). Here tı is the 
training data point, and condition- 
ing on the value of tı, correspond- 
ing to the vertical blue line, we ob- 
tain p(t2|t1) shown as a function of 5 
t2 by the green curve. 
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framework. However, an advantage of a Gaussian processes viewpoint is that we 
can consider covariance functions that can only be expressed in terms of an infinite 
number of basis functions. 

For large training data sets, however, the direct application of Gaussian process 
methods can become infeasible, and so a range of approximation schemes have been 
developed that have better scaling with training set size than the exact approach 
(Gibbs, 1997; Tresp, 2001; Smola and Bartlett, 2001; Williams and Seeger, 2001; 
Csat6 and Opper, 2002; Seeger et al., 2003). Practical issues in the application of 
Gaussian processes are discussed in Bishop and Nabney (2008). 

We have introduced Gaussian process regression for the case of a single tar- 
get variable. The extension of this formalism to multiple target variables, known 
as co-kriging (Cressie, 1993), is straightforward. Various other extensions of Gaus- 


Illustration of Gaussian process re- 
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data set in Figure A.6 in which the 1f (0) 1 
three right-most data points have 
been omitted. The green curve 95} 
shows the sinusoidal function from ce) 

which the data points, shown in o\ O 
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sian process regression have also been considered, for purposes such as modelling 
the distribution over low-dimensional manifolds for unsupervised learning (Bishop 
et al., 1998a) and the solution of stochastic differential equations (Graepel, 2003). 


6.4.3 Learning the hyperparameters 


The predictions of a Gaussian process model will depend, in part, on the choice 
of covariance function. In practice, rather than fixing the covariance function, we 
may prefer to use a parametric family of functions and then infer the parameter 
values from the data. These parameters govern such things as the length scale of the 
correlations and the precision of the noise and correspond to the hyperparameters in 
a standard parametric model. 

Techniques for learning the hyperparameters are based on the evaluation of the 
likelihood function p(t|@) where @ denotes the hyperparameters of the Gaussian pro- 
cess model. The simplest approach is to make a point estimate of 0 by maximizing 
the log likelihood function. Because 0 represents a set of hyperparameters for the 
regression problem, this can be viewed as analogous to the type 2 maximum like- 
lihood procedure for linear regression models. Maximization of the log likelihood 
can be done using efficient gradient-based optimization algorithms such as conjugate 
gradients (Fletcher, 1987; Nocedal and Wright, 1999; Bishop and Nabney, 2008). 

The log likelihood function for a Gaussian process regression model is easily 
evaluated using the standard form for a multivariate Gaussian distribution, giving 


1 1 N 
Inp(t|@) = —5 In [Cy] — zt Ont — 5 n@7). (6.69) 


For nonlinear optimization, we also need the gradient of the log likelihood func- 
tion with respect to the parameter vector 8. We shall assume that evaluation of the 
derivatives of C y is straightforward, as would be the case for the covariance func- 
tions considered in this chapter. Making use of the result (C.21) for the derivative of 
Cy together with the result (C.22) for the derivative of In |C y|, we obtain 


ð aul e a , Up 28 44 
zg, npttl®) = -3T (cx a) Eat OM pent (6.70) 


Because In p(t|@) will in general be a nonconvex function, it can have multiple max- 
ima. 

It is straightforward to introduce a prior over 0 and to maximize the log poste- 
rior using gradient-based methods. In a fully Bayesian treatment, we need to evaluate 
marginals over 0 weighted by the product of the prior p(@) and the likelihood func- 
tion p(t|@). In general, however, exact marginalization will be intractable, and we 
must resort to approximations. 

The Gaussian process regression model gives a predictive distribution whose 
mean and variance are functions of the input vector x. However, we have assumed 
that the contribution to the predictive variance arising from the additive noise, gov- 
erned by the parameter (3, is a constant. For some problems, known as heteroscedas- 
tic, the noise variance itself will also depend on x. To model this, we can extend the 
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Figure 6.9 Samples from the ARD 


prior for Gaussian processes, in 
which the kernel function is given by 
(6.71). The left plot corresponds to 
™ = 2 = 1, and the right plot cor- 
responds to 7 = 1, 72 = 0.01. 


Gaussian process framework by introducing a second Gaussian process to represent 
the dependence of ( on the input x (Goldberg et al., 1998). Because (3 is a variance, 
and hence nonnegative, we use the Gaussian process to model In G(x). 


6.4.4 Automatic relevance determination 


In the previous section, we saw how maximum likelihood could be used to de- 
termine a value for the correlation length-scale parameter in a Gaussian process. 
This technique can usefully be extended by incorporating a separate parameter for 
each input variable (Rasmussen and Williams, 2006). The result, as we shall see, is 
that the optimization of these parameters by maximum likelihood allows the relative 
importance of different inputs to be inferred from the data. This represents an exam- 
ple in the Gaussian process context of automatic relevance determination, or ARD, 
which was originally formulated in the framework of neural networks (MacKay, 
1994; Neal, 1996). The mechanism by which appropriate inputs are preferred is 
discussed in Section 7.2.2. 

Consider a Gaussian process with a two-dimensional input space x = (21, £2), 
having a kernel function of the form 


2 
k(x, x’) = 0o exp f- S nl: — “yt f (6.71) 
i=1 


Samples from the resulting prior over functions y(x) are shown for two different 
settings of the precision parameters 7; in Figure 6.9. We see that, as a particu- 
lar parameter 7; becomes small, the function becomes relatively insensitive to the 
corresponding input variable x;. By adapting these parameters to a data set using 
maximum likelihood, it becomes possible to detect input variables that have little 
effect on the predictive distribution, because the corresponding values of 7; will be 
small. This can be useful in practice because it allows such inputs to be discarded. 
ARD is illustrated using a simple synthetic data set having three inputs £1, £2 and x3 
(Nabney, 2002) in Figure 6.10. The target variable t, is generated by sampling 100 
values of zı from a Gaussian, evaluating the function sin(272,), and then adding 
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Illustration of automatic rele- 
vance determination in a Gaus- 
sian process for a synthetic prob- 
lem having three inputs x1, x2, 
and x3, for which the curves 
show the corresponding values of 
the hyperparameters 7; (red), n2 
(green), and 73 (blue) as a func- 
tion of the number of iterations 
when optimizing the marginal 
likelihood. Details are given in 
the text. Note the logarithmic 
scale on the vertical axis. 


0 20 40 60 80 100 


Gaussian noise. Values of x2 are given by copying the corresponding values of xı 
and adding noise, and values of x3 are sampled from an independent Gaussian dis- 
tribution. Thus 2, is a good predictor of t, £ is a more noisy predictor of t, and x3 
has only chance correlations with t. The marginal likelihood for a Gaussian process 
with ARD parameters 7, 72,73 is optimized using the scaled conjugate gradients 
algorithm. We see from Figure 6.10 that 7; converges to a relatively large value, 12 
converges to a much smaller value, and 73 becomes very small indicating that x3 is 
irrelevant for predicting t. 

The ARD framework is easily incorporated into the exponential-quadratic kernel 
(6.63) to give the following form of kernel function, which has been found useful for 
applications of Gaussian processes to a range of regression problems 


D D 
1 
k(Xn, Xm) = bo exp -3 3 Mil Tri = Emi) + 02 + 03 se Lnikmi (6.72) 


i=1 i=1 
where D is the dimensionality of the input space. 


6.4.5 Gaussian processes for classification 


In a probabilistic approach to classification, our goal is to model the posterior 
probabilities of the target variable for a new input vector, given a set of training 
data. These probabilities must lie in the interval (0,1), whereas a Gaussian process 
model makes predictions that lie on the entire real axis. However, we can easily 
adapt Gaussian processes to classification problems by transforming the output of 
the Gaussian process using an appropriate nonlinear activation function. 

Consider first the two-class problem with a target variable t € {0,1}. If we de- 
fine a Gaussian process over a function a(x) and then transform the function using 
a logistic sigmoid y = o(a), given by (4.59), then we will obtain a non-Gaussian 
stochastic process over functions y(x) where y € (0,1). This is illustrated for the 
case of a one-dimensional input space in Figure 6.11 in which the probability distri- 
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Figure 6.11 The left plot shows a sample from a Gaussian process prior over functions a(x), and the right plot 
shows the result of transforming this sample using a logistic sigmoid function. 


bution over the target variable t is then given by the Bernoulli distribution 


p(tla) = o(a)*(1 — o(a))**. (6.73) 
As usual, we denote the training set inputs by x1,...,x, with corresponding 
observed target variables t = (t,,...,t,)?. We also consider a single test point 


Xn41 With target value ty41. Our goal is to determine the predictive distribution 
p(tn+i\t), where we have left the conditioning on the input variables implicit. To do 
this we introduce a Gaussian process prior over the vector ay +1, which has compo- 
nents a(x1),...,@(X+1). This in turn defines a non-Gaussian process over tv +1, 
and by conditioning on the training data t y we obtain the required predictive distri- 
bution. The Gaussian process prior for ay, takes the form 


p(anyi) = N(an4i|0, Cy 41). (6.74) 


Unlike the regression case, the covariance matrix no longer includes a noise term 
because we assume that all of the training data points are correctly labelled. How- 
ever, for numerical reasons it is convenient to introduce a noise-like term governed 
by a parameter v that ensures that the covariance matrix is positive definite. Thus 
the covariance matrix C y+; has elements given by 


C(Xn, Xm) = k(Xn, Xm) + Vônm (6.75) 


where k(Xn, Xm) is any positive semidefinite kernel function of the kind considered 
in Section 6.2, and the value of v is typically fixed in advance. We shall assume that 
the kernel function k(x, x’) is governed by a vector 0 of parameters, and we shall 
later discuss how @ may be learned from the training data. 

For two-class problems, it is sufficient to predict p(ty41 = 1|ty) because the 
value of p(tn+1 = Olt) is then given by 1 — p(tw41 = 1|\tw). The required 
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predictive distribution is given by 
P(twa1 = 1|tw) = icon = llan41)p(anyiltw) dan 41 (6.76) 


where p(tn41 = llan4i1) = o(an41). 

This integral is analytically intractable, and so may be approximated using sam- 
pling methods (Neal, 1997). Alternatively, we can consider techniques based on 
an analytical approximation. In Section 4.5.2, we derived the approximate formula 
(4.153) for the convolution of a logistic sigmoid with a Gaussian distribution. We 
can use this result to evaluate the integral in (6.76) provided we have a Gaussian 
approximation to the posterior distribution p(ay +,|ty). The usual justification for a 
Gaussian approximation to a posterior distribution is that the true posterior will tend 
to a Gaussian as the number of data points increases as a consequence of the central 
limit theorem. In the case of Gaussian processes, the number of variables grows with 
the number of data points, and so this argument does not apply directly. However, if 
we consider increasing the number of data points falling in a fixed region of x space, 
then the corresponding uncertainty in the function a(x) will decrease, again leading 
asymptotically to a Gaussian (Williams and Barber, 1998). 

Three different approaches to obtaining a Gaussian approximation have been 
considered. One technique is based on variational inference (Gibbs and MacKay, 
2000) and makes use of the local variational bound (10.144) on the logistic sigmoid. 
This allows the product of sigmoid functions to be approximated by a product of 
Gaussians thereby allowing the marginalization over ay to be performed analyti- 
cally. The approach also yields a lower bound on the likelihood function p(t y |). 
The variational framework for Gaussian process classification can also be extended 
to multiclass (K > 2) problems by using a Gaussian approximation to the softmax 
function (Gibbs, 1997). 

A second approach uses expectation propagation (Opper and Winther, 2000b; 
Minka, 2001b; Seeger, 2003). Because the true posterior distribution is unimodal, as 
we shall see shortly, the expectation propagation approach can give good results. 


6.4.6 Laplace approximation 


The third approach to Gaussian process classification is based on the Laplace 
approximation, which we now consider in detail. In order to evaluate the predictive 
distribution (6.76), we seek a Gaussian approximation to the posterior distribution 
over ay+1, which, using Bayes’ theorem, is given by 


p(ansiltw) = J vlaws,anitw) daw 


1 
~ a | romanojn an) daw 
1 


= sity | Maw-alaw plan )pltylan) dax 


[vlan lan )p(an|tn) day (6.77) 
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where we have used p(twlan+1,an) = p(twlany). The conditional distribution 
p(an+i|an) is obtained by invoking the results (6.66) and (6.67) for Gaussian pro- 
cess regression, to give 


P(an4ilanw) = N(an4i|k' Cy an, ¢ —k™Cj'k). (6.78) 


We can therefore evaluate the integral in (6.77) by finding a Laplace approximation 
for the posterior distribution p(ay|ty), and then using the standard result for the 
convolution of two Gaussian distributions. 

The prior p(ay) is given by a zero-mean Gaussian process with covariance ma- 
trix Cy, and the data term (assuming independence of the data points) is given by 


p(tvlan) = IIo y'(1 — o(an))*~ "=I - antn o(—an). (6.79) 


We then obtain the Laplace approximation by Taylor expanding the logarithm of 
p(an|ty), which up to an additive normalization constant is given by the quantity 


V(ay) = Inp(ay) + Inp(ty|an) 


1 N 1 
—=ayCy an — —In(2r) — =In|Cy| +tyan 

2 2 2 
=Y in(1 + e%") + const. (6.80) 


First we need to find the mode of the posterior distribution, and this requires that we 
evaluate the gradient of Y (ayn), which is given by 


VU(ay) =ty — on — Cyan (6.81) 


where oy is a vector with elements o(a,,). We cannot simply find the mode by 
setting this gradient to zero, because oy depends nonlinearly on ay, and so we 
resort to an iterative scheme based on the Newton-Raphson method, which gives rise 
to an iterative reweighted least squares (IRLS) algorithm. This requires the second 
derivatives of Y (ay), which we also require for the Laplace approximation anyway, 
and which are given by 


VVU(an) = -Wn -Cp (6.82) 


where W y is a diagonal matrix with elements o(a,,)(1— o(a,,)), and we have used 
the result (4.88) for the derivative of the logistic sigmoid function. Note that these 
diagonal elements lie in the range (0,1/4), and hence Wy is a positive definite 
matrix. Because C y (and hence its inverse) is positive definite by construction, and 
because the sum of two positive definite matrices is also positive definite, we see 
that the Hessian matrix A = —VVW(ay.) is positive definite and so the posterior 
distribution p(ay|ty) is log convex and therefore has a single mode that is the global 
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maximum. The posterior distribution is not Gaussian, however, because the Hessian 
is a function of ay. 

Using the Newton-Raphson formula (4.92), the iterative update equation for ay 
is given by 


ay” = Cy(I+ WyCy)* {tn -on + Wynan}. (6.83) 


These equations are iterated until they converge to the mode which we denote by 
ax. At the mode, the gradient VY (ay) will vanish, and hence a, will satisfy 


any = Cy(ty — on). (6.84) 


Once we have found the mode a7, of the posterior, we can evaluate the Hessian 
matrix given by 
H = -VVW(ay) =Wywt+Cy (6.85) 


where the elements of W y are evaluated using aġ. This defines our Gaussian ap- 
proximation to the posterior distribution p(ay|ty ) given by 


q(an) =Nilaylay,H *). (6.86) 


We can now combine this with (6.78) and hence evaluate the integral (6.77). Because 
this corresponds to a linear-Gaussian model, we can use the general result (2.115) to 
give 


Elanyilty] = k'(ty —on) (6.87) 
var[anqiltw] = c—k™(Wy' 4+ Cw) tk. (6.88) 


Now that we have a Gaussian distribution for p(ay+1|ty), we can approximate 
the integral (6.76) using the result (4.153). As with the Bayesian logistic regression 
model of Section 4.5, if we are only interested in the decision boundary correspond- 
ing to p(ty+ıltn) = 0.5, then we need only consider the mean and we can ignore 
the effect of the variance. 

We also need to determine the parameters 0 of the covariance function. One 
approach is to maximize the likelihood function given by p(ty |0) for which we need 
expressions for the log likelihood and its gradient. If desired, suitable regularization 
terms can also be added, leading to a penalized maximum likelihood solution. The 
likelihood function is defined by 


p(tn|@) = J vttwiaw law) daw. (6.89) 


This integral is analytically intractable, so again we make use of the Laplace approx- 
imation. Using the result (4.135), we obtain the following approximation for the log 
of the likelihood function 


1 N 
In p(tw|6) = Y(aġ) — 5 In| Ww + Cy |+ z m2r) (6.90) 
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where U(at,) = In p(aġ |0) + Inp(ty|a%,). We also need to evaluate the gradient 
of In p(t~|@) with respect to the parameter vector 0. Note that changes in @ will 
cause changes in a}, leading to additional terms in the gradient. Thus, when we 
differentiate (6.90) with respect to 8, we obtain two sets of terms, the first arising 
from the dependence of the covariance matrix Cy on 0, and the rest arising from 
dependence of az, on 8. 

The terms arising from the explicit dependence on @ can be found by using 
(6.80) together with the results (C.21) and (C.22), and are given by 


On p(tn|4) Le ai CN ais 
a0; g9N Cn “99, CN AN 
1 
== it (I+ Cy Wy)! Wy 2OX . (6.91) 
2 30; 


To compute the terms arising from the dependence of a, on 0, we note that 
the Laplace approximation has been constructed such that (axy) has zero gradient 
at ay = aj, and so W(a‘,) gives no contribution to the gradient as a result of its 
dependence on aj,. This leaves the following contribution to the derivative with 
respect to a component 0; of 0 


N 


1 5 ôln |W n + Ci’ | ax 
2 = daž 00; 
I< dax 
D =1 * * * n 
=-5 D [I+ CyvWy) Ca], a o} — 20%) 36, (692) 
where ož = o(a*), and again we have used the result (C.22) together with the 


definition of W y. We can evaluate the derivative of a}, with respect to 0; by differ- 
entiating the relation (6.84) with respect to 0; to give 


Oa* OCNn Oa* 

a0; a0, (ty — on) nWw 0, (6.93) 
Rearranging then gives 

Oar _,0CN 

30; ~ (I+ WnCn) 20, (ty — ow). (6.94) 


Combining (6.91), (6.92), and (6.94), we can evaluate the gradient of the log 
likelihood function, which can be used with standard nonlinear optimization algo- 
rithms in order to determine a value for 8. 

We can illustrate the application of the Laplace approximation for Gaussian pro- 
cesses using the synthetic two-class data set shown in Figure 6.12. Extension of the 
Laplace approximation to Gaussian processes involving K > 2 classes, using the 
softmax activation function, is straightforward (Williams and Barber, 1998). 
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Figure 6.12 Illustration of the use of a Gaussian process for classification, showing the data on the left together 
with the optimal decision boundary from the true distribution in green, and the decision boundary from the 
Gaussian process classifier in black. On the right is the predicted posterior probability for the blue and red 
classes together with the Gaussian process decision boundary. 


6.4.7 Connection to neural networks 


We have seen that the range of functions which can be represented by a neural 
network is governed by the number M of hidden units, and that, for sufficiently 
large M, a two-layer network can approximate any given function with arbitrary 
accuracy. In the framework of maximum likelihood, the number of hidden units 
needs to be limited (to a level dependent on the size of the training set) in order 
to avoid over-fitting. However, from a Bayesian perspective it makes little sense to 
limit the number of parameters in the network according to the size of the training 
set. 

In a Bayesian neural network, the prior distribution over the parameter vector 
w, in conjunction with the network function f(x, w), produces a prior distribution 
over functions from y(x) where y is the vector of network outputs. Neal (1996) 
has shown that, for a broad class of prior distributions over w, the distribution of 
functions generated by a neural network will tend to a Gaussian process in the limit 
M — oo. It should be noted, however, that in this limit the output variables of the 
neural network become independent. One of the great merits of neural networks is 
that the outputs share the hidden units and so they can ‘borrow statistical strength’ 
from each other, that is, the weights associated with each hidden unit are influenced 
by all of the output variables not just by one of them. This property is therefore lost 
in the Gaussian process limit. 

We have seen that a Gaussian process is determined by its covariance (kernel) 
function. Williams (1998) has given explicit forms for the covariance in the case of 
two specific choices for the hidden unit activation function (probit and Gaussian). 
These kernel functions k(x, x’) are nonstationary, i.e. they cannot be expressed as 
a function of the difference x — x’, as a consequence of the Gaussian weight prior 
being centred on zero which breaks translation invariance in weight space. 
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6.4 


6.5 
6.6 
6.7 
6.8 
6.9 
6.10 


By working directly with the covariance function we have implicitly marginal- 
ized over the distribution of weights. If the weight prior is governed by hyperpa- 
rameters, then their values will determine the length scales of the distribution over 
functions, as can be understood by studying the examples in Figure 5.11 for the case 
of a finite number of hidden units. Note that we cannot marginalize out the hyperpa- 
rameters analytically, and must instead resort to techniques of the kind discussed in 
Section 6.4. 


œ» FQ Consider the dual formulation of the least squares linear regression 
problem given in Section 6.1. Show that the solution for the components a,, of 
the vector a can be expressed as a linear combination of the elements of the vector 
(xX,,). Denoting these coefficients by the vector w, show that the dual of the dual 
formulation is given by the original representation in terms of the parameter vector 
w. 


(xx) In this exercise, we develop a dual formulation of the perceptron learning 
algorithm. Using the perceptron learning rule (4.55), show that the learned weight 
vector w can be written as a linear combination of the vectors t,@(x,,) where t, € 
{—1,+1}. Denote the coefficients of this linear combination by a,, and derive a 
formulation of the perceptron learning algorithm, and the predictive function for the 
perceptron, in terms of the a,,. Show that the feature vector @(x) enters only in the 
form of the kernel function k(x, x’) = 6(x)T(x’). 


(x) The nearest-neighbour classifier (Section 2.5.2) assigns a new input vector x 
to the same class as that of the nearest input vector x, from the training set, where 
in the simplest case, the distance is defined by the Euclidean metric ||x — x,,||?. By 
expressing this rule in terms of scalar products and then making use of kernel sub- 
stitution, formulate the nearest-neighbour classifier for a general nonlinear kernel. 


(x) In Appendix C, we give an example of a matrix that has positive elements but 
that has a negative eigenvalue and hence that is not positive definite. Find an example 
of the converse property, namely a 2 x 2 matrix with positive eigenvalues yet that 
has at least one negative element. 


(x) | www | Verify the results (6.13) and (6.14) for constructing valid kernels. 
(x) Verify the results (6.15) and (6.16) for constructing valid kernels. 
(x) | www | Verify the results (6.17) and (6.18) for constructing valid kernels. 
(x) Verify the results (6.19) and (6.20) for constructing valid kernels. 
(x) Verify the results (6.21) and (6.22) for constructing valid kernels. 


(x) Show that an excellent choice of kernel for learning a function f(x) is given 
by k(x, x’) = f(x) f(x’) by showing that a linear learning machine based on this 
kernel will always find a solution proportional to f(x). 
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(x) By making use of the expansion (6.25), and then expanding the middle factor 
as a power series, show that the Gaussian kernel (6.23) can be expressed as the inner 
product of an infinite-dimensional feature vector. 


œ» EA Consider the space of all possible subsets A of a given fixed set D. 
Show that the kernel function (6.27) corresponds to an inner product in a feature 
space of dimensionality 2!! defined by the mapping (A) where A is a subset of D 
and the element ġy (A), indexed by the subset U, is given by 


1, ifU CA; 
pu(A)= { 0, otherwise. (699) 


Here U C A denotes that U is either a subset of A or is equal to A. 


(x) Show that the Fisher kernel, defined by (6.33), remains invariant if we make 
a nonlinear transformation of the parameter vector 9 — (8), where the function 
¢(-) is invertible and differentiable. 


œ FY Write down the form of the Fisher kernel, defined by (6.33), for the 
case of a distribution p(x|u) = N (x| u, S) that is Gaussian with mean p and fixed 
covariance S. 


(x) By considering the determinant of a 2 x 2 Gram matrix, show that a positive- 
definite kernel function k(x, x’) satisfies the Cauchy-Schwartz inequality 


k(a1, £2)? < k(x1, 21)k(x2, £2). (6.96) 


(xx) Consider a parametric model governed by the parameter vector w together 
with a data set of input values x,,...,x, and a nonlinear feature mapping (x). 
Suppose that the dependence of the error function on w takes the form 


JI(w) = f(w' o(x1),...,w' @(xn)) + g(w'w) (6.97) 


where g(-) is a monotonically increasing function. By writing w in the form 


N 
w= > AnP(Xn) +W (6.98) 


n=1 


show that the value of w that minimizes J (w) takes the form of a linear combination 
of the basis functions @(x,,) for n = 1,..., N. 


œx» EA Consider the sum-of-squares error function (6.39) for data having 
noisy inputs, where 1(€) is the distribution of the noise. Use the calculus of vari- 
ations to minimize this error function with respect to the function y(x), and hence 
show that the optimal solution is given by an expansion of the form (6.40) in which 
the basis functions are given by (6.41). 
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6.24 


(x) Consider a Nadaraya-Watson model with one input variable x and one target 
variable t having Gaussian components with isotropic covariances, so that the co- 
variance matrix is given by o?I where I is the unit matrix. Write down expressions 
for the conditional density p(t|x) and for the conditional mean E[t|x] and variance 
var|t|a], in terms of the kernel function k(x, £n). 


(xx) Another viewpoint on kernel regression comes from a consideration of re- 
gression problems in which the input variables as well as the target variables are 
corrupted with additive noise. Suppose each target value t,, is generated as usual 
by taking a function y(z,,) evaluated at a point Zn, and adding Gaussian noise. The 
value of z,, is not directly observed, however, but only a noise corrupted version 
Xn = Zn + €,, where the random variable € is governed by some distribution g(§). 
Consider a set of observations {Xņ, tn}, where n = 1,..., N, together with a cor- 
responding sum-of-squares error function defined by averaging over the distribution 
of input noise to give 


1 N 
=> >, i {y(&n — En) — tn} 9(En) dên- (6.99) 


By minimizing E with respect to the function y(z) using the calculus of variations 
(Appendix D), show that optimal solution for y(x) is given by a Nadaraya-Watson 
kernel regression solution of the form (6.45) with a kernel of the form (6.46). 


(x x) UY Verify the results (6.66) and (6.67). 


œ» FQ Consider a Gaussian process regression model in which the kernel 
function is defined in terms of a fixed set of nonlinear basis functions. Show that the 
predictive distribution is identical to the result (3.58) obtained in Section 3.3.2 for the 
Bayesian linear regression model. To do this, note that both models have Gaussian 
predictive distributions, and so it is only necessary to show that the conditional mean 
and variance are the same. For the mean, make use of the matrix identity (C.6), and 
for the variance, make use of the matrix identity (C.7). 


(xx) Consider a regression problem with N training set input vectors x;,...,xXN 
and L test set input vectors xy41,...,XN+z, and suppose we define a Gaussian 
process prior over functions t(x). Derive an expression for the joint predictive dis- 
tribution for t(xy+1), -- - , t(XN+L), given the values of t(x,),...,¢(xy). Show the 
marginal of this distribution for one of the test observations t; where N+ 1 < j < 
N + Lis given by the usual Gaussian process regression result (6.66) and (6.67). 


œ» PQQ Consider a Gaussian process regression model in which the target 
variable t has dimensionality D. Write down the conditional distribution of t N+1 
for a test input vector xv +1, given a training set of input vectors X1, ...,Xy+1 and 
corresponding target observations t,,..., ty. 


(x) Show that a diagonal matrix W whose elements satisfy 0 < W;; < 1 is positive 
definite. Show that the sum of two positive definite matrices is itself positive definite. 
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œ [QTY Using the Newton-Raphson formula (4.92), derive the iterative update 
formula (6.83) for finding the mode a‘, of the posterior distribution in the Gaussian 
process classification model. 


(x) Using the result (2.115), derive the expressions (6.87) and (6.88) for the mean 
and variance of the posterior distribution p(ay+1|ty) in the Gaussian process clas- 
sification model. 


(xxx) Derive the result (6.90) for the log likelihood function in the Laplace approx- 
imation framework for Gaussian process classification. Similarly, derive the results 
(6.91), (6.92), and (6.94) for the terms in the gradient of the log likelihood. 


In the previous chapter, we explored a variety of learning algorithms based on non- 
linear kernels. One of the significant limitations of many such algorithms is that 
the kernel function k(Xn, Xm) must be evaluated for all possible pairs x,, and xm 
of training points, which can be computationally infeasible during training and can 
lead to excessive computation times when making predictions for new data points. 
In this chapter we shall look at kernel-based algorithms that have sparse solutions, 
so that predictions for new inputs depend only on the kernel function evaluated at a 
subset of the training data points. 

We begin by looking in some detail at the support vector machine (SVM), which 
became popular in some years ago for solving problems in classification, regression, 
and novelty detection. An important property of support vector machines is that the 
determination of the model parameters corresponds to a convex optimization prob- 
lem, and so any local solution is also a global optimum. Because the discussion of 
support vector machines makes extensive use of Lagrange multipliers, the reader is 
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encouraged to review the key concepts covered in Appendix E. Additional infor- 
mation on support vector machines can be found in Vapnik (1995), Burges (1998), 
Cristianini and Shawe-Taylor (2000), Muller et al. (2001), Scholkopf and Smola 
(2002), and Herbrich (2002). 

The SVM is a decision machine and so does not provide posterior probabilities. 
We have already discussed some of the benefits of determining probabilities in Sec- 
tion 1.5.4. An alternative sparse kernel technique, known as the relevance vector 
machine (RVM), is based on a Bayesian formulation and provides posterior proba- 
bilistic outputs, as well as having typically much sparser solutions than the SVM. 


Maximum Margin Classifiers 


We begin our discussion of support vector machines by returning to the two-class 
classification problem using linear models of the form 


y(x) = w' d(x) +b (7.1) 


where (x) denotes a fixed feature-space transformation, and we have made the 
bias parameter b explicit. Note that we shall shortly introduce a dual representation 
expressed in terms of kernel functions, which avoids having to work explicitly in 
feature space. The training data set comprises N input vectors x,,...,xy, with 
corresponding target values t1, ...,ty where t,, € {—1,1}, and new data points x 
are classified according to the sign of y(x). 

We shall assume for the moment that the training data set is linearly separable in 
feature space, so that by definition there exists at least one choice of the parameters 
w and b such that a function of the form (7.1) satisfies y(x,,) > 0 for points having 
tn = +1 and y(xn) < 0 for points having tn = —1, so that tny(Xn) > 0 for all 
training data points. 

There may of course exist many such solutions that separate the classes exactly. 
In Section 4.1.7, we described the perceptron algorithm that is guaranteed to find 
a solution in a finite number of steps. The solution that it finds, however, will be 
dependent on the (arbitrary) initial values chosen for w and b as well as on the 
order in which the data points are presented. If there are multiple solutions all of 
which classify the training data set exactly, then we should try to find the one that 
will give the smallest generalization error. The support vector machine approaches 
this problem through the concept of the margin, which is defined to be the smallest 
distance between the decision boundary and any of the samples, as illustrated in 
Figure 7.1. 

In support vector machines the decision boundary is chosen to be the one for 
which the margin is maximized. The maximum margin solution can be motivated us- 
ing computational learning theory, also known as statistical learning theory. How- 
ever, a simple insight into the origins of maximum margin has been given by Tong 
and Koller (2000) who consider a framework for classification based on a hybrid of 
generative and discriminative approaches. They first model the distribution over in- 
put vectors x for each class using a Parzen density estimator with Gaussian kernels 
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margin 


Figure 7.1 The margin is defined as the perpendicular distance between the decision boundary and the closest 
of the data points, as shown on the left figure. Maximizing the margin leads to a particular choice of decision 
boundary, as shown on the right. The location of this boundary is determined by a subset of the data points, 
known as support vectors, which are indicated by the circles. 


having a common parameter o°. Together with the class priors, this defines an opti- 
mal misclassification-rate decision boundary. However, instead of using this optimal 
boundary, they determine the best hyperplane by minimizing the probability of error 
relative to the learned density model. In the limit c? — 0, the optimal hyperplane 
is shown to be the one having maximum margin. The intuition behind this result is 
that as a° is reduced, the hyperplane is increasingly dominated by nearby data points 
relative to more distant ones. In the limit, the hyperplane becomes independent of 
data points that are not support vectors. 

We shall see in Figure 10.13 that marginalization with respect to the prior distri- 
bution of the parameters in a Bayesian approach for a simple linearly separable data 
set leads to a decision boundary that lies in the middle of the region separating the 
data points. The large margin solution has similar behaviour. 

Recall from Figure 4.1 that the perpendicular distance of a point x from a hyper- 
plane defined by y(x) = 0 where y(x) takes the form (7.1) is given by |y(x)|/||w|. 
Furthermore, we are only interested in solutions for which all data points are cor- 
rectly classified, so that t,,y(x,,) > 0 for all n. Thus the distance of a point x, to the 
decision surface is given by 


try(Xn) _ tn(wib(xn) +b) (7.2) 


The margin is given by the perpendicular distance to the closest point x,, from the 
data set, and we wish to optimize the parameters w and b in order to maximize this 
distance. Thus the maximum margin solution is found by solving 


arg pmax | Iw] min lt ñ (w (xn) + a} (7.3) 


where we have taken the factor 1/||w|| outside the optimization over n because w 
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does not depend on n. Direct solution of this optimization problem would be very 
complex, and so we shall convert it into an equivalent problem that is much easier 
to solve. To do this we note that if we make the rescaling w — «xw and b — kb, 
then the distance from any point Xn to the decision surface, given by tny(Xn)/||w 
is unchanged. We can use this freedom to set 


tn (w'P(Xn) +b) =1 (7.4) 


for the point that is closest to the surface. In this case, all data points will satisfy the 
constraints 


>, 


tn (w' (xn) +b) > 1, n=1,..., N. (1.5) 


This is known as the canonical representation of the decision hyperplane. In the 
case of data points for which the equality holds, the constraints are said to be active, 
whereas for the remainder they are said to be inactive. By definition, there will 
always be at least one active constraint, because there will always be a closest point, 
and once the margin has been maximized there will be at least two active constraints. 
The optimization problem then simply requires that we maximize ||w||~', which is 
equivalent to minimizing ||w||*, and so we have to solve the optimization problem 


1 
arg min = ||w]|? (7.6) 
w,b 2 


subject to the constraints given by (7.5). The factor of 1/2 in (7.6) is included for 
later convenience. This is an example of a quadratic programming problem in which 
we are trying to minimize a quadratic function subject to a set of linear inequality 
constraints. It appears that the bias parameter b has disappeared from the optimiza- 
tion. However, it is determined implicitly via the constraints, because these require 
that changes to ||w|| be compensated by changes to b. We shall see how this works 
shortly. 

In order to solve this constrained optimization problem, we introduce Lagrange 
multipliers an > 0, with one multiplier an for each of the constraints in (7.5), giving 
the Lagrangian function 


N 
1 
L(w, b,a) = ziw? — p an {tn(w" (Xn) + 6) — 1} (7.7) 
n=1 
where a = (a;,...,a@y)7. Note the minus sign in front of the Lagrange multiplier 


term, because we are minimizing with respect to w and b, and maximizing with 
respect to a. Setting the derivatives of L(w, b,a) with respect to w and b equal to 
zero, we obtain the following two conditions 


N 


w = So antr$(Xn) (7.8) 


n=1 
N 

0 = Sle (7.9) 
n=} 
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Eliminating w and b from L(w,b,a) using these conditions then gives the dual 
representation of the maximum margin problem in which we maximize 


N 
L(a) = dam — S35 Gndiralnlrnd En Xm) (7.10) 


see e 


with respect to a subject to the constraints 


an > 0, n=1,...,N, (7.11) 


N 
Sate = (7.12) 
n=1 


Here the kernel function is defined by k(x, x’) = @(x)"@(x’). Again, this takes the 
form of a quadratic programming problem in which we optimize a quadratic function 
of a subject to a set of inequality constraints. We shall discuss techniques for solving 
such quadratic programming problems in Section 7.1.1. 

The solution to a quadratic programming problem in M variables in general has 
computational complexity that is O( M°). In going to the dual formulation we have 
turned the original optimization problem, which involved minimizing (7.6) over M 
variables, into the dual problem (7.10), which has N variables. For a fixed set of 
basis functions whose number M is smaller than the number N of data points, the 
move to the dual problem appears disadvantageous. However, it allows the model to 
be reformulated using kernels, and so the maximum margin classifier can be applied 
efficiently to feature spaces whose dimensionality exceeds the number of data points, 
including infinite feature spaces. The kernel formulation also makes clear the role 
of the constraint that the kernel function k(x, x’) be positive definite, because this 
ensures that the Lagrangian function L(a) is bounded below, giving rise to a well- 
defined optimization problem. 

In order to classify new data points using the trained model, we evaluate the sign 
of y(x) defined by (7.1). This can be expressed in terms of the parameters {an } and 
the kernel function by substituting for w using (7.8) to give 


ESN X, Xn) +0. (7.13) 


n=1 


! Joseph-Louis Lagrange years, Euler worked hard to persuade Lagrange to 
1736-1813 move to Berlin, which he eventually did in 1766 where 

he succeeded Euler as Director of Mathematics at 
Although widely considered to be the Berlin Academy. Later he moved to Paris, nar- 
a French mathematician, Lagrange rowly escaping with his life during the French revo- 
was born in Turin in Italy. By the age lution thanks to the personal intervention of Lavoisier 
of nineteen, he had already made (the French chemist who discovered oxygen) who him- 
important contributions mathemat- self was later executed at the guillotine. Lagrange 
ics and had been appointed as Pro- made key contributions to the calculus of variations 
fessor at the Royal Artillery School in Turin. For many and the foundations of dynamics. 
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In Appendix E, we show that a constrained optimization of this form satisfies the 
Karush-Kuhn-Tucker (KKT) conditions, which in this case require that the following 
three properties hold 


an > 0 (7.14) 
tny(Xn) -1 > 0 (7.15) 
an {tny(xXn)—- 1} = 0: (7.16) 


Thus for every data point, either an = 0 or tny(Xn) = 1. Any data point for 
which an = 0 will not appear in the sum in (7.13) and hence plays no role in making 
predictions for new data points. The remaining data points are called support vectors, 
and because they satisfy t,y(x,) = 1, they correspond to points that lie on the 
maximum margin hyperplanes in feature space, as illustrated in Figure 7.1. This 
property is central to the practical applicability of support vector machines. Once 
the model is trained, a significant proportion of the data points can be discarded and 
only the support vectors retained. 

Having solved the quadratic programming problem and found a value for a, we 
can then determine the value of the threshold parameter b by noting that any support 
vector X,, satisfies t,,y(x,,) = 1. Using (7.13) this gives 


tn (= amtmk(Xn, Xm) + ) =1 (7.17) 


mes 


where S denotes the set of indices of the support vectors. Although we can solve 
this equation for b using an arbitrarily chosen support vector xn, a numerically more 
stable solution is obtained by first multiplying through by tn, making use of t? = 1, 
and then averaging these equations over all support vectors and solving for b to give 


1 
b= Ns D (e — 5. ee) (7.18) 


mes 


where Ns is the total number of support vectors. 

For later comparison with alternative models, we can express the maximum- 
margin classifier in terms of the minimization of an error function, with a simple 
quadratic regularizer, in the form 


N 
S > Boo (y(%n)tn — 1) + Allwll? (7.19) 


n=1 


where E.(z) is a function that is zero if z > 0 and oo otherwise and ensures that 
the constraints (7.5) are satisfied. Note that as long as the regularization parameter 
satisfies \ > 0, its precise value plays no role. 

Figure 7.2 shows an example of the classification resulting from training a sup- 
port vector machine on a simple synthetic data set using a Gaussian kernel of the 
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Figure 7.2 Example of synthetic data from 
two classes in two dimensions 
showing contours of constant 
y(x) obtained from a support 
vector machine having a Gaus- 
sian kernel function. Also shown 
are the decision boundary, the 
margin boundaries, and the sup- 
port vectors. 


form (6.23). Although the data set is not linearly separable in the two-dimensional 
data space x, it is linearly separable in the nonlinear feature space defined implicitly 
by the nonlinear kernel function. Thus the training data points are perfectly separated 
in the original data space. 

This example also provides a geometrical insight into the origin of sparsity in 
the SVM. The maximum margin hyperplane is defined by the location of the support 
vectors. Other data points can be moved around freely (so long as they remain out- 
side the margin region) without changing the decision boundary, and so the solution 
will be independent of such data points. 


7.1.1 Overlapping class distributions 


So far, we have assumed that the training data points are linearly separable in the 
feature space @(x). The resulting support vector machine will give exact separation 
of the training data in the original input space x, although the corresponding decision 
boundary will be nonlinear. In practice, however, the class-conditional distributions 
may overlap, in which case exact separation of the training data can lead to poor 
generalization. 

We therefore need a way to modify the support vector machine so as to allow 
some of the training points to be misclassified. From (7.19) we see that in the case 
of separable classes, we implicitly used an error function that gave infinite error 
if a data point was misclassified and zero error if it was classified correctly, and 
then optimized the model parameters to maximize the margin. We now modify this 
approach so that data points are allowed to be on the ‘wrong side’ of the margin 
boundary, but with a penalty that increases with the distance from that boundary. For 
the subsequent optimization problem, it is convenient to make this penalty a linear 
function of this distance. To do this, we introduce slack variables, £, > 0 where 
n = 1,...,N, with one slack variable for each training data point (Bennett, 1992; 
Cortes and Vapnik, 1995). These are defined by €,, = 0 for data points that are on or 
inside the correct margin boundary and €,, = |tn — y(x»)| for other points. Thus a 
data point that is on the decision boundary y(x,,) = 0 will have €,, = 1, and points 
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Figure 7.3 


Illustration of the slack variables €, > 0. 
Data points with circles around them are 
support vectors. 


with €,, > 1 will be misclassified. The exact classification constraints (7.5) are then 
replaced with 
tny(Xn) Z 1— £n, m= Ae N. (7.20) 


in which the slack variables are constrained to satisfy €,, > 0. Data points for which 
En = 0 are correctly classified and are either on the margin or on the correct side 
of the margin. Points for which 0 < €, < 1 lie inside the margin, but on the cor- 
rect side of the decision boundary, and those data points for which €, > 1 lie on 
the wrong side of the decision boundary and are misclassified, as illustrated in Fig- 
ure 7.3. This is sometimes described as relaxing the hard margin constraint to give a 
soft margin and allows some of the training set data points to be misclassified. Note 
that while slack variables allow for overlapping class distributions, this framework is 
still sensitive to outliers because the penalty for misclassification increases linearly 
with €. 

Our goal is now to maximize the margin while softly penalizing points that lie 
on the wrong side of the margin boundary. We therefore minimize 


N 
1 
Cy int zwi? (1.21) 
n=1 


where the parameter C > 0 controls the trade-off between the slack variable penalty 
and the margin. Because any point that is misclassified has £» > 1, it follows that 
Xn Ên is an upper bound on the number of misclassified points. The parameter C is 
therefore analogous to (the inverse of) a regularization coefficient because it controls 
the trade-off between minimizing training errors and controlling model complexity. 
In the limit C — oo, we will recover the earlier support vector machine for separable 
data. 

We now wish to minimize (7.21) subject to the constraints (7.20) together with 
En 2 0. The corresponding Lagrangian is given by 


N 
L(w,b,a) = = ilw roy 6 so {tny(Xn) -1 + én} Y Anén (7.22) 


n=1 n=1 
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where {an > 0} and {un > 0} are Lagrange multipliers. The corresponding set of 
KKT conditions are given by 


Qn > O (7.23) 

tny(Xn) -1+f, > 0 (7.24) 
an (tny(Xn) -1+&)) = 0 (7.25) 
lin > 0 (7.26) 

Ena > 0 (7.27) 

ie, = Ú (7.28) 


where n = 1,..., N. 
We now optimize out w, b, and {€,,} making use of the definition (7.1) of y(x) 
to give 


ðL y 
a 0 > w= >, Onin @(Xn) (7.29) 
N 

OL 
a 0 > >, antn = 0 (7.30) 
OL 

2 ee 731 
dE, 0 > a C-u (7.31) 


Using these results to eliminate w, b, and {€,,} from the Lagrangian, we obtain the 
dual Lagrangian in the form 


7 N i 
L(a) = Qn — 5 5 > Clint lvl Xn Xm) (7.32) 


n=1 n=1m=1 


which is identical to the separable case, except that the constraints are somewhat 
different. To see what these constraints are, we note that a, > 0 is required because 
these are Lagrange multipliers. Furthermore, (7.31) together with un > 0 implies 
Gn < C. We therefore have to minimize (7.32) with respect to the dual variables 
{an } subject to 


0O<an<C (7.33) 
N 
> Gin =0 (7.34) 
n=1 
for n = 1,..., N, where (7.33) are known as box constraints. This again represents 


a quadratic programming problem. If we substitute (7.29) into (7.1), we see that 
predictions for new data points are again made by using (7.13). 

We can now interpret the resulting solution. As before, a subset of the data 
points may have an = 0, in which case they do not contribute to the predictive 
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model (7.13). The remaining data points constitute the support vectors. These have 
an > 0 and hence from (7.25) must satisfy 


tny(Xn) = 1 — ĉn- (7.35) 


If an < C, then (7.31) implies that un > 0, which from (7.28) requires €,, = 0 and 
hence such points lie on the margin. Points with a,, = C can lie inside the margin 
and can either be correctly classified if €,, < 1 or misclassified if é» > 1. 

To determine the parameter b in (7.1), we note that those support vectors for 
which 0 < an < C have £n = 0 so that tny(Xn) = 1 and hence will satisfy 


tn (= amtmk(Xn, Xm) + ) =1. (7.36) 


mes 


Again, a numerically stable solution is obtained by averaging to give 


b= Ne 5 (+ = oe) (7.37) 


meS 


where M denotes the set of indices of data points having 0 < a, < C. 
An alternative, equivalent formulation of the support vector machine, known as 
the v-SVM, has been proposed by Scholkopf et al. (2000). This involves maximizing 


Jo 1 N N 
L(a) = 9 5 5 anümtnimk(Xn, Xr) (7.38) 


n=1m=1 


subject to the constraints 


0<an<1/N (7.39) 
N 
Sante =0 (7.40) 
n=1 
N 
Yon > v. (7.41) 
m=i 


This approach has the advantage that the parameter v, which replaces C, can be 
interpreted as both an upper bound on the fraction of margin errors (points for which 
En > 0 and hence which lie on the wrong side of the margin boundary and which may 
or may not be misclassified) and a lower bound on the fraction of support vectors. An 
example of the v-SVM applied to a synthetic data set is shown in Figure 7.4. Here 
Gaussian kernels of the form exp (—y||x — x’ ||?) have been used, with y = 0.45. 
Although predictions for new inputs are made using only the support vectors, 
the training phase (i.e., the determination of the parameters a and b) makes use of 
the whole data set, and so it is important to have efficient algorithms for solving 
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Illustration of the v-SVM applied 


to a nonseparable data set in two ® 
dimensions. The support vectors 2| 
are indicated by circles. 
x 
0 l 
—2+ 
-2 


the quadratic programming problem. We first note that the objective function L(a) 
given by (7.10) or (7.32) is quadratic and so any local optimum will also be a global 
optimum provided the constraints define a convex region (which they do as a conse- 
quence of being linear). Direct solution of the quadratic programming problem us- 
ing traditional techniques is often infeasible due to the demanding computation and 
memory requirements, and so more practical approaches need to be found. The tech- 
nique of chunking (Vapnik, 1982) exploits the fact that the value of the Lagrangian 
is unchanged if we remove the rows and columns of the kernel matrix corresponding 
to Lagrange multipliers that have value zero. This allows the full quadratic pro- 
gramming problem to be broken down into a series of smaller ones, whose goal is 
eventually to identify all of the nonzero Lagrange multipliers and discard the others. 
Chunking can be implemented using protected conjugate gradients (Burges, 1998). 
Although chunking reduces the size of the matrix in the quadratic function from the 
number of data points squared to approximately the number of nonzero Lagrange 
multipliers squared, even this may be too big to fit in memory for large-scale appli- 
cations. Decomposition methods (Osuna et al., 1996) also solve a series of smaller 
quadratic programming problems but are designed so that each of these is of a fixed 
size, and so the technique can be applied to arbitrarily large data sets. However, it 
still involves numerical solution of quadratic programming subproblems and these 
can be problematic and expensive. One of the most popular approaches to training 
support vector machines is called sequential minimal optimization, or SMO (Platt, 
1999). It takes the concept of chunking to the extreme limit and considers just two 
Lagrange multipliers at a time. In this case, the subproblem can be solved analyti- 
cally, thereby avoiding numerical quadratic programming altogether. Heuristics are 
given for choosing the pair of Lagrange multipliers to be considered at each step. 
In practice, SMO is found to have a scaling with the number of data points that is 
somewhere between linear and quadratic depending on the particular application. 
We have seen that kernel functions correspond to inner products in feature spaces 
that can have high, or even infinite, dimensionality. By working directly in terms of 
the kernel function, without introducing the feature space explicitly, it might there- 
fore seem that support vector machines somehow manage to avoid the curse of di- 
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mensionality. This is not the case, however, because there are constraints amongst 
the feature values that restrict the effective dimensionality of feature space. To see 
this consider a simple second-order polynomial kernel that we can expand in terms 
of its components 


(1 + xa) = (1 + £121 + £222) 

= 14 24,2, + 2T222 4 rz? + 2£1Z21£2Z2 + eee 

(1, V221, V229, 07, V20129, x3)(1, V 221, V222, 22, V22122, zaj 
p(x)" p(z). (7.42) 


This kernel function therefore represents an inner product in a feature space having 
six dimensions, in which the mapping from input space to feature space is described 
by the vector function @(x). However, the coefficients weighting these different 
features are constrained to have specific forms. Thus any set of points in the original 
two-dimensional space x would be constrained to lie exactly on a two-dimensional 
nonlinear manifold embedded in the six-dimensional feature space. 

We have already highlighted the fact that the support vector machine does not 
provide probabilistic outputs but instead makes classification decisions for new in- 
put vectors. Veropoulos et al. (1999) discuss modifications to the SVM to allow 
the trade-off between false positive and false negative errors to be controlled. How- 
ever, if we wish to use the SVM as a module in a larger probabilistic system, then 
probabilistic predictions of the class label t for new inputs x are required. 

To address this issue, Platt (2000) has proposed fitting a logistic sigmoid to the 
outputs of a previously trained support vector machine. Specifically, the required 
conditional probability is assumed to be of the form 


k(x, 2) 


p(t = 1|x) = o (Ay(x) + B) (7.43) 


where y(x) is defined by (7.1). Values for the parameters A and B are found by 
minimizing the cross-entropy error function defined by a training set consisting of 
pairs of values y(x,,) and tn. The data used to fit the sigmoid needs to be independent 
of that used to train the original SVM in order to avoid severe over-fitting. This two- 
stage approach is equivalent to assuming that the output y(x) of the support vector 
machine represents the log-odds of x belonging to class t = 1. Because the SVM 
training procedure is not specifically intended to encourage this, the SVM can give 
a poor approximation to the posterior probabilities (Tipping, 2001). 


7.1.2 Relation to logistic regression 


As with the separable case, we can re-cast the SVM for nonseparable distri- 
butions in terms of the minimization of a regularized error function. This will also 
allow us to highlight similarities, and differences, compared to the logistic regression 
model. 

We have seen that for data points that are on the correct side of the margin 
boundary, and which therefore satisfy ynt, > 1, we have én = 0, and for the 
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Figure 7.5 Plot of the ‘hinge’ error function used E(z) 


Exercise 7.6 


in support vector machines, shown 
in blue, along with the error function 
for logistic regression, rescaled by a 
factor of 1/1n(2) so that it passes 
through the point (0, 1), shown in red. 
Also shown are the misclassification 
error in black and the squared error 
in green. 
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remaining points we have €,, = 1 — Yntn. Thus the objective function (7.21) can be 
written (up to an overall multiplicative constant) in the form 


N 
X Esv (yntan) + Allwll? (7.44) 
m=i 


where A = (2C)~', and Esy(-) is the hinge error function defined by 
Esv(yntn) = (1 = Yntn] y (7.45) 


where [-]+ denotes the positive part. The hinge error function, so-called because 
of its shape, is plotted in Figure 7.5. It can be viewed as an approximation to the 
misclassification error, i.e., the error function that ideally we would like to minimize, 
which is also shown in Figure 7.5. 

When we considered the logistic regression model in Section 4.3.2, we found it 
convenient to work with target variable t € {0,1}. For comparison with the support 
vector machine, we first reformulate maximum likelihood logistic regression using 
the target variable t € {—1,1}. To do this, we note that p(t = 1|y) = o(y) where 
y(x) is given by (7.1), and o(y) is the logistic sigmoid function defined by (4.59). It 
follows that p(t = —1|y) = 1 — o(y) = o(—y), where we have used the properties 
of the logistic sigmoid function, and so we can write 


p(tly) = o(yt). (7.46) 


From this we can construct an error function by taking the negative logarithm of the 
likelihood function that, with a quadratic regularizer, takes the form 


N 
XO Exa(Yntn) + Alfwll?. (1.47) 
n=1 


where 
Exr(yt) = ln (1 + exp(—yt)) . (7.48) 
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For comparison with other error functions, we can divide by In(2) so that the error 
function passes through the point (0,1). This rescaled error function is also plotted 
in Figure 7.5 and we see that it has a similar form to the support vector error function. 
The key difference is that the flat region in Esv (yt) leads to sparse solutions. 

Both the logistic error and the hinge loss can be viewed as continuous approx- 
imations to the misclassification error. Another continuous error function that has 
sometimes been used to solve classification problems is the squared error, which 
is again plotted in Figure 7.5. It has the property, however, of placing increasing 
emphasis on data points that are correctly classified but that are a long way from 
the decision boundary on the correct side. Such points will be strongly weighted at 
the expense of misclassified points, and so if the objective is to minimize the mis- 
classification rate, then a monotonically decreasing error function would be a better 
choice. 


7.1.3 Multiclass SVMs 


The support vector machine is fundamentally a two-class classifier. In practice, 
however, we often have to tackle problems involving K > 2 classes. Various meth- 
ods have therefore been proposed for combining multiple two-class SVMs in order 
to build a multiclass classifier. 

One commonly used approach (Vapnik, 1998) is to construct K separate SVMs, 
in which the kt® model y;,(x) is trained using the data from class Cx as the positive 
examples and the data from the remaining K — 1 classes as the negative examples. 
This is known as the one-versus-the-rest approach. However, in Figure 4.2 we saw 
that using the decisions of the individual classifiers can lead to inconsistent results 
in which an input is assigned to multiple classes simultaneously. This problem is 
sometimes addressed by making predictions for new inputs x using 


y(x) = MAX Yi (x): (7.49) 


Unfortunately, this heuristic approach suffers from the problem that the different 
classifiers were trained on different tasks, and there is no guarantee that the real- 
valued quantities y(x) for different classifiers will have appropriate scales. 

Another problem with the one-versus-the-rest approach is that the training sets 
are imbalanced. For instance, if we have ten classes each with equal numbers of 
training data points, then the individual classifiers are trained on data sets comprising 
90% negative examples and only 10% positive examples, and the symmetry of the 
original problem is lost. A variant of the one-versus-the-rest scheme was proposed 
by Lee et al. (2001) who modify the target values so that the positive class has target 
+1 and the negative class has target —1/(K — 1). 

Weston and Watkins (1999) define a single objective function for training all 
K SVMs simultaneously, based on maximizing the margin from each to remaining 
classes. However, this can result in much slower training because, instead of solving 
K separate optimization problems each over N data points with an overall cost of 
O(K N°), a single optimization problem of size (K — 1) N must be solved giving an 
overall cost of O( K? N?). 
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Another approach is to train K (K — 1) /2 different 2-class SVMs on all possible 
pairs of classes, and then to classify test points according to which class has the high- 
est number of ‘votes’, an approach that is sometimes called one-versus-one. Again, 
we saw in Figure 4.2 that this can lead to ambiguities in the resulting classification. 
Also, for large K this approach requires significantly more training time than the 
one-versus-the-rest approach. Similarly, to evaluate test points, significantly more 
computation is required. 

The latter problem can be alleviated by organizing the pairwise classifiers into 
a directed acyclic graph (not to be confused with a probabilistic graphical model) 
leading to the DAGSVM (Platt et al., 2000). For K classes, the DAGSVM has a total 
of K(k — 1)/2 classifiers, and to classify a new test point only K — 1 pairwise 
classifiers need to be evaluated, with the particular classifiers used depending on 
which path through the graph is traversed. 

A different approach to multiclass classification, based on error-correcting out- 
put codes, was developed by Dietterich and Bakiri (1995) and applied to support 
vector machines by Allwein et al. (2000). This can be viewed as a generalization of 
the voting scheme of the one-versus-one approach in which more general partitions 
of the classes are used to train the individual classifiers. The K classes themselves 
are represented as particular sets of responses from the two-class classifiers chosen, 
and together with a suitable decoding scheme, this gives robustness to errors and to 
ambiguity in the outputs of the individual classifiers. Although the application of 
SVMs to multiclass classification problems remains an open issue, in practice the 
one-versus-the-rest approach is the most widely used in spite of its ad-hoc formula- 
tion and its practical limitations. 

There are also single-class support vector machines, which solve an unsuper- 
vised learning problem related to probability density estimation. Instead of mod- 
elling the density of data, however, these methods aim to find a smooth boundary 
enclosing a region of high density. The boundary is chosen to represent a quantile of 
the density, that is, the probability that a data point drawn from the distribution will 
land inside that region is given by a fixed number between 0 and 1 that is specified in 
advance. This is a more restricted problem than estimating the full density but may 
be sufficient in specific applications. Two approaches to this problem using support 
vector machines have been proposed. The algorithm of Scholkopf et al. (2001) tries 
to find a hyperplane that separates all but a fixed fraction v of the training data from 
the origin while at the same time maximizing the distance (margin) of the hyperplane 
from the origin, while Tax and Duin (1999) look for the smallest sphere in feature 
space that contains all but a fraction v of the data points. For kernels k(x, x’) that 
are functions only of x — x’, the two algorithms are equivalent. 


7.1.4 SVMs for regression 


We now extend support vector machines to regression problems while at the 
same time preserving the property of sparseness. In simple linear regression, we 
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Figure 7.6 Plot of an e-insensitive error function (in 


7. SPARSE KERNEL MACHINES 


red) in which the error increases lin- 
early with distance beyond the insen- 
sitive region. Also shown for compar- 
ison is the quadratic error function (in 
green). 


minimize a regularized error function given by 


ie À 
5 in- tn}? + Sllwll?. (7.50) 
n=1 


To obtain sparse solutions, the quadratic error function is replaced by an €-insensitive 
error function (Vapnik, 1995), which gives zero error if the absolute difference be- 
tween the prediction y(x) and the target t is less than € where « > 0. A simple 
example of an €-insensitive error function, having a linear cost associated with errors 
outside the insensitive region, is given by 


Bee = =| ee ee (7.51) 
and is illustrated in Figure 7.6. 
We therefore minimize a regularized error function given by 
2 1 
CX E.(y(&n) — tn) + ziw? (1.52) 


where y(x) is given by (7.1). By convention the (inverse) regularization parameter, 
denoted C', appears in front of the error term. 

As before, we can re-express the optimization problem by introducing slack 
variables. For each data point xn, we now need two slack variables €,, > 0 and 
Èn > > 0, where ¿n > 0 corresponds to a point for which tn > y(Xn) + €, and En >0 
corresponds to a point for which tn < y(Xn) — €, as illustrated in Figure 7.7. 

The condition for a target point to lie inside the e-tube is that yn — € < tn < 
Yn +€, where yn = y(X,,). Introducing the slack variables allows points to lie outside 
the tube provided the slack variables are nonzero, and the corresponding conditions 
are 


th < yl(xn)te+h&, (7.53) 
tn & ye, )-e— ĉn. (7.54) 
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Figure 7.7 Illustration of SVM regression, showing 


the regression curve together with the e- y(x) yte 
insensitive ‘tube’. Also shown are exam- £>0 y 
ples of the slack variables ¢ and €. Points 
above the «-tube have € > 0 and € = 0, y—e 
points below the e-tube have € = 0 and ‘ 
E > 0, and points inside the «-tube have 
—€-0. a 
=e >o 
£ 


The error function for support vector regression can then be written as 


N 


COG +&) E (1.55) 


n=1 


which must be minimized subject to the constraints én > 0 and En > 0 as well as 
(7.53) and (7.54). This can be achieved by introducing Lagrange multipliers an > 0, 
Gn > 0, Un > 0, and ün > 0 and optimizing the Lagrangian 


N = 1 N PA 
b= OY katai ziw? RCs re) 
n=1 n=1 


N N 
-X ane + &a + Un — tn) — X Gn(€ + En — Yn ttn). (7.56) 
n=1 


We now substitute for y(x) using (7.1) and then set the derivatives of the La- 
grangian with respect to w, b, €,, and €,, to zero, giving 


N 
OL A 
Iw =0 > w= Ah = Gn) (Xn) (7.57) 
N 
OL & 
a os 2 (an —@n)=0 (7.58) 
ðL 
a = 0 > antptn=C (7.59) 
L A a 
a =0 => GQatin=C. (7.60) 
OEn 


Using these results to eliminate the corresponding variables from the Lagrangian, we 
Exercise 7.7 see that the dual problem involves maximizing 
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N 
—e X (an + Gn) + X (an — ûn)tn (7.61) 


with respect to {an} and {a,,}, where we have introduced the kernel k(x,x’) = 
o(x)' d(x’). Again, this is a constrained maximization, and to find the constraints 
we note that an > 0 and a, > 0 are both required because these are Lagrange 
multipliers. Also un > 0 and ji, > O together with (7.59) and (7.60), require 
an < C and a, < C, and so again we have the box constraints 


0Sa SC (7.62) 
(7,20 (7.63) 


together with the condition (7.58). 

Substituting (7.57) into (7.1), we see that predictions for new inputs can be made 
using 

N 
y(x) = Sp — Gn)k(x,Xn) +b (7.64) 
n=1 

which is again expressed in terms of the kernel function. 

The corresponding Karush-Kuhn-Tucker (KKT) conditions, which state that at 
the solution the product of the dual variables and the constraints must vanish, are 
given by 


An(€ + En + Yn — tn) 0 (7.65) 
Gn(€ +n —Ynttn) = 0 (7.66) 
(C—an)&, = 0 (7.67) 
(C—Gn)ér = 0 (7.68) 


From these we can obtain several useful results. First of all, we note that a coefficient 
an can only be nonzero if € + En + Yn — tn = 0, which implies that the data point 
either lies on the upper boundary of the e-tube (n = 0) or lies above the upper 
boundary (€,, > 0). Similarly, a nonzero value for @,, implies € + Ên — Yn + tn = 0, 
and such points must lie either on or below the lower boundary of the e-tube. 

Furthermore, the two constraints € + En + Yn — tn = 0 and e€ + En — Yn +tn =0 
are incompatible, as is easily seen by adding them together and noting that n» and 
En are nonnegative while e€ is strictly positive, and so for every data point x,,, either 
an Or Gy, (or both) must be zero. 

The support vectors are those data points that contribute to predictions given by 
(7.64), in other words those for which either a,, 4 0 or a@,, 4 0. These are points that 
lie on the boundary of the ¢-tube or outside the tube. All points within the tube have 
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an = ûn = 0. We again have a sparse solution, and the only terms that have to be 
evaluated in the predictive model (7.64) are those that involve the support vectors. 

The parameter b can be found by considering a data point for which 0 < an < 
C, which from (7.67) must have én = 0, and from (7.65) must therefore satisfy 
€+ Yn — tn = 0. Using (7.1) and solving for b, we obtain 


b = tr—e—w' (xn) 
N 
= Pes S Gn — Gm)k(Xn,Xm) (7.69) 
m=1 


where we have used (7.57). We can obtain an analogous result by considering a point 
for which 0 < a, < C. In practice, it is better to average over all such estimates of 
b. 

As with the classification case, there is an alternative formulation of the SVM 
for regression in which the parameter governing complexity has a more intuitive 
interpretation (Scholkopf et al., 2000). In particular, instead of fixing the width € of 
the insensitive region, we fix instead a parameter v that bounds the fraction of points 
lying outside the tube. This involves maximizing 


n=1m=1 
N 
(i enh (7.70) 
n=1 
subject to the constraints 
0 < an <S C/N (7.71) 
0 < @n < C/N (7.72) 
N 
Y (nia) =0 (7.73) 
n=1 
N 
S (an 4 Gn) < UC; (7.74) 
n=1 


It can be shown that there are at most vN data points falling outside the insensitive 
tube, while at least vN data points are support vectors and so lie either on the tube 
or outside it. 

The use of a support vector machine to solve a regression problem is illustrated 
using the sinusoidal data set in Figure 7.8. Here the parameters v and C have been 
chosen by hand. In practice, their values would typically be determined by cross- 
validation. 
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Figure 7.8 


Illustration of the v-SVM for re- 
gression applied to the sinusoidal 
synthetic data set using Gaussian if o © 
kernels. The predicted regression 

curve is shown by the red line, and + 
the c-insensitive tube corresponds © 

to the shaded region. Also, the ORO) © 
data points are shown in green, Of ] 
and those with support vectors 
are indicated by blue circles. © 


7.1.5 Computational learning theory 


Historically, support vector machines have largely been motivated and analysed 
using a theoretical framework known as computational learning theory, also some- 
times called statistical learning theory (Anthony and Biggs, 1992; Kearns and Vazi- 
rani, 1994; Vapnik, 1995; Vapnik, 1998). This has its origins with Valiant (1984) 
who formulated the probably approximately correct, or PAC, learning framework. 
The goal of the PAC framework is to understand how large a data set needs to be in 
order to give good generalization. It also gives bounds for the computational cost of 
learning, although we do not consider these here. 

Suppose that a data set D of size N is drawn from some joint distribution p(x, t) 
where x is the input variable and t represents the class label, and that we restrict 
attention to ‘noise free’ situations in which the class labels are determined by some 
(unknown) deterministic function t = g(x). In PAC learning we say that a function 
f(x; D), drawn from a space F of such functions on the basis of the training set 
D, has good generalization if its expected error rate is below some pre-specified 
threshold €, so that 


Eze l (El DVA) << (7.75) 


where J(-) is the indicator function, and the expectation is with respect to the dis- 
tribution p(x,t). The quantity on the left-hand side is a random variable, because 
it depends on the training set D, and the PAC framework requires that (7.75) holds, 
with probability greater than 1 — ô, for a data set D drawn randomly from p(x, t). 
Here 6 is another pre-specified parameter, and the terminology ‘probably approxi- 
mately correct’ comes from the requirement that with high probability (greater than 
1 — ô), the error rate be small (less than €). For a given choice of model space F, and 
for given parameters € and 6, PAC learning aims to provide bounds on the minimum 
size N of data set needed to meet this criterion. A key quantity in PAC learning is 
the Vapnik-Chervonenkis dimension, or VC dimension, which provides a measure of 
the complexity of a space of functions, and which allows the PAC framework to be 
extended to spaces containing an infinite number of functions. 

The bounds derived within the PAC framework are often described as worst- 
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case, because they apply to any choice for the distribution p(x, t), so long as both 
the training and the test examples are drawn (independently) from the same distribu- 
tion, and for any choice for the function f(x) so long as it belongs to F. In real-world 
applications of machine learning, we deal with distributions that have significant reg- 
ularity, for example in which large regions of input space carry the same class label. 
As a consequence of the lack of any assumptions about the form of the distribution, 
the PAC bounds are very conservative, in other words they strongly over-estimate 
the size of data sets required to achieve a given generalization performance. For this 
reason, PAC bounds have found few, if any, practical applications. 

One attempt to improve the tightness of the PAC bounds is the PAC-Bayesian 
framework (McAllester, 2003), which considers a distribution over the space F of 
functions, somewhat analogous to the prior in a Bayesian treatment. This still con- 
siders any possible choice for p(x, t), and so although the bounds are tighter, they 
are still very conservative. 


Relevance Vector Machines 


Support vector machines have been used in a variety of classification and regres- 
sion applications. Nevertheless, they suffer from a number of limitations, several 
of which have been highlighted already in this chapter. In particular, the outputs of 
an SVM represent decisions rather than posterior probabilities. Also, the SVM was 
originally formulated for two classes, and the extension to K > 2 classes is prob- 
lematic. There is a complexity parameter C’, or v (as well as a parameter € in the case 
of regression), that must be found using a hold-out method such as cross-validation. 
Finally, predictions are expressed as linear combinations of kernel functions that are 
centred on training data points and that are required to be positive definite. 

The relevance vector machine or RVM (Tipping, 2001) is a Bayesian sparse ker- 
nel technique for regression and classification that shares many of the characteristics 
of the SVM whilst avoiding its principal limitations. Additionally, it typically leads 
to much sparser models resulting in correspondingly faster performance on test data 
whilst maintaining comparable generalization error. 

In contrast to the SVM we shall find it more convenient to introduce the regres- 
sion form of the RVM first and then consider the extension to classification tasks. 


7.2.1 RVM for regression 


The relevance vector machine for regression is a linear model of the form studied 
in Chapter 3 but with a modified prior that results in sparse solutions. The model 
defines a conditional distribution for a real-valued target variable t, given an input 
vector x, which takes the form 


p(t|x, w, 8) = N(tly(x), 8) (7.76) 
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where 8 = o~? is the noise precision (inverse noise variance), and the mean is given 
by a linear model of the form 


M 
u(x) = $ wigi(x) = wT bx) (7.77) 


with fixed nonlinear basis functions ¢;(x), which will typically include a constant 
term so that the corresponding weight parameter represents a ‘bias’. 

The relevance vector machine is a specific instance of this model, which is in- 
tended to mirror the structure of the support vector machine. In particular, the basis 
functions are given by kernels, with one kernel associated with each of the data 
points from the training set. The general expression (7.77) then takes the SVM-like 
form 


N 
y(x) = > Wnk(X, Xn) +b (7.78) 
n=1 


where b is a bias parameter. The number of parameters in this case is M = N + 1, 
and y(x) has the same form as the predictive model (7.64) for the SVM, except that 
the coefficients a,, are here denoted wn. It should be emphasized that the subsequent 
analysis is valid for arbitrary choices of basis function, and for generality we shall 
work with the form (7.77). In contrast to the SVM, there is no restriction to positive- 
definite kernels, nor are the basis functions tied in either number or location to the 
training data points. 

Suppose we are given a set of N observations of the input vector x, which we 
denote collectively by a data matrix X whose n'" row is x! with n = 1,...,.N. The 
corresponding target values are given by t = (t,,...,t))*. Thus, the likelihood 
function is given by 


N 


p(t|X, w, 8) = | | plénlxn, w, 67>). (7.79) 


n=1 


Next we introduce a prior distribution over the parameter vector w and as in 
Chapter 3, we shall consider a zero-mean Gaussian prior. However, the key differ- 
ence in the RVM is that we introduce a separate hyperparameter œ; for each of the 
weight parameters w; instead of a single shared hyperparameter. Thus the weight 
prior takes the form 


M 
pwle) = |] V(wil0,a;7) (7.80) 
$=] 


where a; represents the precision of the corresponding parameter w;, and œ denotes 
(a1,...,@,7)". We shall see that, when we maximize the evidence with respect 
to these hyperparameters, a significant proportion of them go to infinity, and the 
corresponding weight parameters have posterior distributions that are concentrated 
at zero. The basis functions associated with these parameters therefore play no role 
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in the predictions made by the model and so are effectively pruned out, resulting in 
a sparse model. 

Using the result (3.49) for linear regression models, we see that the posterior 
distribution for the weights is again Gaussian and takes the form 


p(wit, X, a, 3) = N(wlm, £) (1.81) 
where the mean and covariance are given by 
m = ~=8"t (7.82) 
E = (A+ 967)" (7.83) 


where ® is the N x M design matrix with elements ®,,; = ¢;(Xn), and A = 
diag(a;). Note that in the specific case of the model (7.78), we have ® = K, where 
K is the symmetric (N + 1) x (N + 1) kernel matrix with elements k(x, Xm). 

The values of œ and 8 are determined using type-2 maximum likelihood, also 
known as the evidence approximation, in which we maximize the marginal likeli- 
hood function obtained by integrating out the weight parameters 


p(t|X, a, B) = J vx. d)p6orla) dw. (7.84) 


Because this represents the convolution of two Gaussians, it is readily evaluated to 
give the log marginal likelihood in the form 
Inp(t|X,a,38) = InN(t0,C) 
1 = 
z {N n(27) + In|C| +C (7.85) 


where t = (tı, ..., ty)", and we have defined the N x N matrix C given by 
C= 67H + PATIST. (7.86) 


Our goal is now to maximize (7.85) with respect to the hyperparameters œ and 
B. This requires only a small modification to the results obtained in Section 3.5 for 
the evidence approximation in the linear regression model. Again, we can identify 
two approaches. In the first, we simply set the required derivatives of the marginal 
likelihood to zero and obtain the following re-estimation equations 


an = a (7.87) 
t— m||? 
ul ie = | m|| (7.88) 


N = DD Vi 


where m; is the it component of the posterior mean m defined by (7.82). The 
quantity y; measures how well the corresponding parameter w; is determined by the 
data and is defined by 
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yi = 1- adi (7.89) 


in which X;; is the i” diagonal component of the posterior covariance © given by 
(7.83). Learning therefore proceeds by choosing initial values for œ and 8, evalu- 
ating the mean and covariance of the posterior using (7.82) and (7.83), respectively, 
and then alternately re-estimating the hyperparameters, using (7.87) and (7.88), and 
re-estimating the posterior mean and covariance, using (7.82) and (7.83), until a suit- 
able convergence criterion is satisfied. 

The second approach is to use the EM algorithm, and is discussed in Sec- 
tion 9.3.4. These two approaches to finding the values of the hyperparameters that 
maximize the evidence are formally equivalent. Numerically, however, it is found 
that the direct optimization approach corresponding to (7.87) and (7.88) gives some- 
what faster convergence (Tipping, 2001). 

As aresult of the optimization, we find that a proportion of the hyperparameters 
{a;i} are driven to large (in principle infinite) values, and so the weight parameters 
w; corresponding to these hyperparameters have posterior distributions with mean 
and variance both zero. Thus those parameters, and the corresponding basis func- 
tions @;(x), are removed from the model and play no role in making predictions for 
new inputs. In the case of models of the form (7.78), the inputs x,, corresponding to 
the remaining nonzero weights are called relevance vectors, because they are iden- 
tified through the mechanism of automatic relevance determination, and are analo- 
gous to the support vectors of an SVM. It is worth emphasizing, however, that this 
mechanism for achieving sparsity in probabilistic models through automatic rele- 
vance determination is quite general and can be applied to any model expressed as 
an adaptive linear combination of basis functions. 

Having found values œ* and {* for the hyperparameters that maximize the 
marginal likelihood, we can evaluate the predictive distribution over t for a new 
input x. Using (7.76) and (7.81), this is given by 


stb: < f p(ilx, w, 8*)p(wlX, t, a*, 8*) dw 


N (t)m™ (x), 0?(x)) . (7.90) 


Thus the predictive mean is given by (7.76) with w set equal to the posterior mean 
m, and the variance of the predictive distribution is given by 


o*(x) = (8) + b(x)* E(x) (7.91) 


where © is given by (7.83) in which œ and {3 are set to their optimized values a* and 
*. This is just the familiar result (3.59) obtained in the context of linear regression. 
Recall that for localized basis functions, the predictive variance for linear regression 
models becomes small in regions of input space where there are no basis functions. 
In the case of an RVM with the basis functions centred on data points, the model will 
therefore become increasingly certain of its predictions when extrapolating outside 
the domain of the data (Rasmussen and Quifionero-Candela, 2005), which of course 
is undesirable. The predictive distribution in Gaussian process regression does not 
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Illustration of RVM regression us- 
ing the same data set, and the 
same Gaussian kernel functions, 
as used in Figure 7.8 for the 
v-SVM regression model. The ¢ 
mean of the predictive distribu- 
tion for the RVM is shown by the 
red line, and the one standard- Of 
deviation predictive distribution is 
shown by the shaded region. 
Also, the data points are shown 
in green, and the relevance vec- -1f 
tors are indicated by blue circles. 
Note that there are only 3 rele- i i 
vance vectors compared to 7 sup- 0 1 
port vectors for the »-SVM in Fig- 
ure 7.8. 


suffer from this problem. However, the computational cost of making predictions 
with a Gaussian processes is typically much higher than with an RVM. 

Figure 7.9 shows an example of the RVM applied to the sinusoidal regression 
data set. Here the noise precision parameter (3 is also determined through evidence 
maximization. We see that the number of relevance vectors in the RVM is signif- 
icantly smaller than the number of support vectors used by the SVM. For a wide 
range of regression and classification tasks, the RVM is found to give models that 
are typically an order of magnitude more compact than the corresponding support 
vector machine, resulting in a significant improvement in the speed of processing on 
test data. Remarkably, this greater sparsity is achieved with little or no reduction in 
generalization error compared with the corresponding SVM. 

The principal disadvantage of the RVM compared to the SVM is that training 
involves optimizing a nonconvex function, and training times can be longer than for a 
comparable SVM. For a model with M basis functions, the RVM requires inversion 
of a matrix of size M x M, which in general requires O( M?) computation. In the 
specific case of the SVM-like model (7.78), we have M = N +1. As we have noted, 
there are techniques for training SVMs whose cost is roughly quadratic in N. Of 
course, in the case of the RVM we always have the option of starting with a smaller 
number of basis functions than N + 1. More significantly, in the relevance vector 
machine the parameters governing complexity and noise variance are determined 
automatically from a single training run, whereas in the support vector machine the 
parameters C and e (or v) are generally found using cross-validation, which involves 
multiple training runs. Furthermore, in the next section we shall derive an alternative 
procedure for training the relevance vector machine that improves training speed 
significantly. 


7.2.2 Analysis of sparsity 


We have noted earlier that the mechanism of automatic relevance determination 
causes a subset of parameters to be driven to zero. We now examine in more detail 
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t2 


Illustration of the mechanism for sparsity in a Bayesian linear regression model, showing a training 


set vector of target values given by t = (ti, t2)", indicated by the cross, for a model with one basis vector 
y = ((x1), 6(x2))", which is poorly aligned with the target data vector t. On the left we see a model having 
only isotropic noise, so that C = 6~'I, corresponding to a = oo, with 3 set to its most probable value. On 
the right we see the same model but with a finite value of a. In each case the red ellipse corresponds to unit 
Mahalanobis distance, with |C| taking the same value for both plots, while the dashed green circle shows the 
contrition arising from the noise term 3~'. We see that any finite value of a reduces the probability of the 
observed data, and so for the most probable solution the basis vector is removed. 


the mechanism of sparsity in the context of the relevance vector machine. In the 
process, we will arrive at a significantly faster procedure for optimizing the hyper- 
parameters compared to the direct techniques given above. 

Before proceeding with a mathematical analysis, we first give some informal 
insight into the origin of sparsity in Bayesian linear models. Consider a data set 
comprising N = 2 observations tı and t2, together with a model having a single 
basis function ¢(x), with hyperparameter a, along with isotropic noise having pre- 
cision 3. From (7.85), the marginal likelihood is given by p(t|a, 3) = N (t|0, C) in 
which the covariance matrix takes the form 


2 
p 


where ~ denotes the N-dimensional vector (¢(x1), ¢(x2))T, and similarly t = 
(tı, t2)". Notice that this is just a zero-mean Gaussian process model over t with 
covariance C. Given a particular observation for t, our goal is to find a* and (* by 
maximizing the marginal likelihood. We see from Figure 7.10 that, if there is a poor 
alignment between the direction of ~ and that of the training data vector t, then the 
corresponding hyperparameter a will be driven to oo, and the basis vector will be 
pruned from the model. This arises because any finite value for a will always assign 
a lower probability to the data, thereby decreasing the value of the density at t, pro- 
vided that ( is set to its optimal value. We see that any finite value for a would cause 
the distribution to be elongated in a direction away from the data, thereby increasing 
the probability mass in regions away from the observed data and hence reducing the 
value of the density at the target data vector itself. For the more general case of M 


1 
C=- -I+ zer" (7.92) 
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basis vectors (,,..., Pay a Similar intuition holds, namely that if a particular basis 
vector is poorly aligned with the data vector t, then it is likely to be pruned from the 
model. 

We now investigate the mechanism for sparsity from a more mathematical per- 
spective, for a general case involving M basis functions. To motivate this analysis 
we first note that, in the result (7.87) for re-estimating the parameter a, the terms on 
the right-hand side are themselves also functions of a;. These results therefore rep- 
resent implicit solutions, and iteration would be required even to determine a single 
a; with all other a; for j Æ i fixed. 

This suggests a different approach to solving the optimization problem for the 
RVM, in which we make explicit all of the dependence of the marginal likelihood 
(7.85) on a particular a; and then determine its stationary points explicitly (Faul and 
Tipping, 2002; Tipping and Faul, 2003). To do this, we first pull out the contribution 
from a; in the matrix C defined by (7.86) to give 


C = I+) oa poj +0797 
JAI 
= Cauta ppr (7.93) 


where q; denotes the it column of ®, in other words the N-dimensional vector with 
elements (@;(x1),..-,;(y)), in contrast to @,,, which denotes the n*™ row of ®. 
The matrix C _; represents the matrix C with the contribution from basis function 7 
removed. Using the matrix identities (C.7) and (C.15), the determinant and inverse 
of C can then be written 


IC| = |C_|[1+ af p; Cr¢,| (7.94) 


co _ Cave 


Cc! : 
“ag + 9F Cz; Hy; 


(7.95) 


Using these results, we can then write the log marginal likelihood function (7.85) in 
the form 
L(a) = L(a_;) + Alai) (7.96) 


where L(œ—;) is simply the log marginal likelihood with basis function y, omitted, 
and the quantity A(a;) is defined by 


1 @ 
Alai) = 7 Ina; — ln (a; + s;) 4 a 


(7.97) 
and contains all of the dependence on a;. Here we have introduced the two quantities 
s = Gi Clip, (7.98) 
qi = pi Cit. (7.99) 


Here s; is called the sparsity and q; is known as the quality of ;, and as we shall 
see, a large value of s; relative to the value of q; means that the basis function »,; 
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Figure 7.11 Plots of the log 


marginal likelihood A(a;) versus ‘ 2 
Ina; showing on the left, the single 
maximum at a finite a; for q? = 4 0 0 
and s; = 1 (so that q? > s;) and on 
the right, the maximum at a; = oo 2 2 
for q? = 1 and s; = 2 (so that 
Gi < si). 

-4 -4 

-5 0 5 -5 0 5 
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is more likely to be pruned from the model. The ‘sparsity’ measures the extent to 
which basis function y; overlaps with the other basis vectors in the model, and the 
‘quality’ represents a measure of the alignment of the basis vector y,, with the error 
between the training set values t = (t,,...,t,’)7 and the vector y_, of predictions 
that would result from the model with the vector y, excluded (Tipping and Faul, 
2003). 

The stationary points of the marginal likelihood with respect to a; occur when 
the derivative 

dX(ai) _ a*s} — (g — si) 
da; 2(a; + s;)? 

is equal to zero. There are two possible forms for the solution. Recalling that a; > 0, 
we see that if q? < si, then a; — œ provides a solution. Conversely, if NA > Si, We 
can solve for a; to obtain 


(7.100) 


a, = =. (7.101) 
qi — Si 
These two solutions are illustrated in Figure 7.11. We see that the relative size of 
the quality and sparsity terms determines whether a particular basis vector will be 
pruned from the model or not. A more complete analysis (Faul and Tipping, 2002), 
based on the second derivatives of the marginal likelihood, confirms these solutions 
are indeed the unique maxima of A(a;). 

Note that this approach has yielded a closed-form solution for a;, for given 
values of the other hyperparameters. As well as providing insight into the origin of 
sparsity in the RVM, this analysis also leads to a practical algorithm for optimizing 
the hyperparameters that has significant speed advantages. This uses a fixed set 
of candidate basis vectors, and then cycles through them in turn to decide whether 
each vector should be included in the model or not. The resulting sequential sparse 
Bayesian learning algorithm is described below. 


Sequential Sparse Bayesian Learning Algorithm 


1. If solving a regression problem, initialize (2. 

2. Initialize using one basis function y,, with hyperparameter a, set using 
(7.101), with the remaining hyperparameters a; for 7 A i initialized to 
infinity, so that only ~, is included in the model. 
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3. Evaluate X and m, along with q; and s; for all basis functions. 
4. Select a candidate basis function ;. 


5. If g > si, and a; < oo, so that the basis vector ọ; is already included in 
the model, then update a; using (7.101). 


6. If g > s;, and a; = oo, then add », to the model, and evaluate hyperpa- 
rameter œ; using (7.101). 


7. If q? < s;, and a; < oo then remove basis function y; from the model, 
and set a; = oo. 


8. If solving a regression problem, update (3. 


9. If converged terminate, otherwise go to 3. 


Note that if q? < s; and a; = ov, then the basis function », is already excluded 
from the model and no action is required. 
In practice, it is convenient to evaluate the quantities 


Qi = yg Ct (7.102) 
Si = giColy,. (7.103) 


The quality and sparseness variables can then be expressed in the form 


aiQi 
e = =e 7.104 
4 Qi = Si ( ) 
aS; 
i = ; 7.105 
5 Qi = Si ( ) 
Note that when a; = 00, we have q; = Q; and s; = S;. Using (C.7), we can write 
Qi = pert- pp SuG't (7.106) 
Si = brp- Ly; GUS" Y, (7.107) 


where ® and & involve only those basis vectors that correspond to finite hyperpa- 
rameters a. At each stage the required computations therefore scale like O( M°), 
where M is the number of active basis vectors in the model and is typically much 
smaller than the number N of training patterns. 


7.2.3 RVM for classification 


We can extend the relevance vector machine framework to classification prob- 
lems by applying the ARD prior over weights to a probabilistic linear classification 
model of the kind studied in Chapter 4. To start with, we consider two-class prob- 
lems with a binary target variable t € {0,1}. The model now takes the form of a 
linear combination of basis functions transformed by a logistic sigmoid function 


y(x,w) = o (w"(x)) (7.108) 
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where o(-) is the logistic sigmoid function defined by (4.59). If we introduce a 
Gaussian prior over the weight vector w, then we obtain the model that has been 
considered already in Chapter 4. The difference here is that in the RVM, this model 
uses the ARD prior (7.80) in which there is a separate precision hyperparameter 
associated with each weight parameter. 

In contrast to the regression model, we can no longer integrate analytically over 
the parameter vector w. Here we follow Tipping (2001) and use the Laplace ap- 
proximation, which was applied to the closely related problem of Bayesian logistic 
regression in Section 4.5.1. 

We begin by initializing the hyperparameter vector œ. For this given value of 
a, we then build a Gaussian approximation to the posterior distribution and thereby 
obtain an approximation to the marginal likelihood. Maximization of this approxi- 
mate marginal likelihood then leads to a re-estimated value for a, and the process is 
repeated until convergence. 

Let us consider the Laplace approximation for this model in more detail. For 
a fixed value of a, the mode of the posterior distribution over w is obtained by 
maximizing 


In p(w|t, œa) = In {p(t]w)p(w|a)} — In p(t|a) 


N 
Í 
= X {tn lnyn + (1 — tn) ln(1 — yn)} — zw Aw +const (7.109) 
n=1 


where A = diag(a;). This can be done using iterative reweighted least squares 
(IRLS) as discussed in Section 4.3.3. For this, we need the gradient vector and 
Hessian matrix of the log posterior distribution, which from (7.109) are given by 


Vinp(wit,a) = &'(t—y)—Aw (7.110) 
VV Inp(wit,a) = -— (TBS +A) (7.111) 


where B is an N x N diagonal matrix with elements bn = y,(1 — yn), the vector 
y = (y1,---, yn) 1, and @ is the design matrix with elements ®,,; = ¢;(x,,). Here 
we have used the property (4.88) for the derivative of the logistic sigmoid function. 
At convergence of the IRLS algorithm, the negative Hessian represents the inverse 
covariance matrix for the Gaussian approximation to the posterior distribution. 

The mode of the resulting approximation to the posterior distribution, corre- 
sponding to the mean of the Gaussian approximation, is obtained setting (7.110) to 
zero, giving the mean and covariance of the Laplace approximation in the form 


w* = A 'l6T(t-y) (7.112) 
S = (TB +A) `. (7.113) 


We can now use this Laplace approximation to evaluate the marginal likelihood. 
Using the general result (4.135) for an integral evaluated using the Laplace approxi- 
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mation, we have 


plia) = f (thw)p(wia) aw 
p(t|w*)p(w* la) (2r). (7.114) 


l2 


If we substitute for p(t|w*) and p(w*|q) and then set the derivative of the marginal 
likelihood with respect to a; equal to zero, we obtain 


(w$)? 4 zu = 0. (7.115) 
Defining y; = 1 — a;%,; and rearranging then gives 


opew a (7.116) 


a 


which is identical to the re-estimation formula (7.87) obtained for the regression 
RVM. 


If we define HN 
t= 6w* +B l(t-y) (7.117) 
we can write the approximate log marginal likelihood in the form 
1 on = 
np(tla, 3) = -3 {N m(27) +m|C|+ Mra) (7.118) 
where 
C=B+6A®’. (7.119) 


This takes the same form as (7.85) in the regression case, and so we can apply the 
same analysis of sparsity and obtain the same fast learning algorithm in which we 
fully optimize a single hyperparameter q; at each step. 

Figure 7.12 shows the relevance vector machine applied to a synthetic classifi- 
cation data set. We see that the relevance vectors tend not to lie in the region of the 
decision boundary, in contrast to the support vector machine. This is consistent with 
our earlier discussion of sparsity in the RVM, because a basis function ¢;(x) centred 
on a data point near the boundary will have a vector ọ; that is poorly aligned with 
the training data vector t. 

One of the potential advantages of the relevance vector machine compared with 
the SVM is that it makes probabilistic predictions. For example, this allows the RVM 
to be used to help construct an emission density in a nonlinear extension of the linear 
dynamical system for tracking faces in video sequences (Williams et al., 2005). 

So far, we have considered the RVM for binary classification problems. For 
K > 2 classes, we again make use of the probabilistic approach in Section 4.3.4 in 
which there are K linear models of the form 


ak = W] X (7.120) 
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Figure 7.12 Example of the relevance vector machine applied to a synthetic data set, in which the left-hand plot 
shows the decision boundary and the data points, with the relevance vectors indicated by circles. Comparison 
with the results shown in Figure 7.4 for the corresponding support vector machine shows that the RVM gives a 
much sparser model. The right-hand plot shows the posterior probability given by the RVM output in which the 
proportion of red (blue) ink indicates the probability of that point belonging to the red (blue) class. 


which are combined using a softmax function to give outputs 


y(x) = p (1.121) 
X exp(a;) 
j 
The log likelihood function is then given by 
N K 
Inp(T|wi,...,wK) = II [ux (7.122) 


where the target values tnx have a 1-of- coding for each data point n, and T is a 
matrix with elements t,,,. Again, the Laplace approximation can be used to optimize 
the hyperparameters (Tipping, 2001), in which the model and its Hessian are found 
using IRLS. This gives a more principled approach to multiclass classification than 
the pairwise method used in the support vector machine and also provides probabilis- 
tic predictions for new data points. The principal disadvantage is that the Hessian 
matrix has size M K x M K, where M is the number of active basis functions, which 
gives an additional factor of K? in the computational cost of training compared with 
the two-class RVM. 

The principal disadvantage of the relevance vector machine is the relatively long 
training times compared with the SVM. This is offset, however, by the avoidance of 
cross-validation runs to set the model complexity parameters. Furthermore, because 
it yields sparser models, the computation time on test points, which is usually the 
more important consideration in practice, is typically much less. 
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(xx) EY Suppose we have a data set of input vectors {xn } with corresponding 
target values t,, € {—1,1}, and suppose that we model the density of input vec- 
tors within each class separately using a Parzen kernel density estimator (see Sec- 
tion 2.5.1) with a kernel k(x, x’). Write down the minimum misclassification-rate 
decision rule assuming the two classes have equal prior probability. Show also that, 
if the kernel is chosen to be k(x, x’) = x? x’, then the classification rule reduces to 
simply assigning a new input vector to the class having the closest mean. Finally, 
show that, if the kernel takes the form k(x, x’) = @(x)"@(x’), that the classification 
is based on the closest mean in the feature space f(x). 


(x) Show that, if the 1 on the right-hand side of the constraint (7.5) is replaced by 
some arbitrary constant y > 0, the solution for the maximum margin hyperplane is 
unchanged. 


(xx) Show that, irrespective of the dimensionality of the data space, a data set 
consisting of just two data points, one from each class, is sufficient to determine the 
location of the maximum-margin hyperplane. 


œ=) FY Show that the value p of the margin for the maximum-margin hyper- 
plane is given by 


N 
= Y ük (7.123) 


where {an} are given by maximizing (7.10) subject to the constraints (7.11) and 
(7.12). 


(xx) Show that the values of p and {an } in the previous exercise also satisfy 


— = 2L(a) (7.124) 


where L(a) is defined by (7.10). Similarly, show that 


1 
— =||\w|l’. (7.125) 


Po 


(x) Consider the logistic regression model with a target variable ¢ € {—1,1}. If 
we define p(t = 1|y) = o(y) where y(x) is given by (7.1), show that the negative 
log likelihood, with the addition of a quadratic regularization term, takes the form 
(7.47). 


(x) Consider the Lagrangian (7.56) for the regression support vector machine. By 
setting the derivatives of the Lagrangian with respect to w, b, En, and En to zero and 
then back substituting to eliminate the corresponding variables, show that the dual 
Lagrangian is given by (7.61). 
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7.8 


7.9 


7.10 


7.11 
7.12 


7.13 


7.14 


7.15 


7.16 


7.17 


7.18 


7.19 


(x) [QQ For the regression support vector machine considered in Section 7.1.4, 
show that all training data points for which ¿n > 0 will have an = C, and similarly 


all points for which En > 0 will have a,, = C. 


(x) Verify the results (7.82) and (7.83) for the mean and covariance of the posterior 
distribution over weights in the regression RVM. 


(« x) www | Derive the result (7.85) for the marginal likelihood function in the 
regression RVM, by performing the Gaussian integral over w in (7.84) using the 
technique of completing the square in the exponential. 


(xx) Repeat the above exercise, but this time make use of the general result (2.115). 


(« x) FQ Show that direct maximization of the log marginal likelihood (7.85) for 
the regression relevance vector machine leads to the re-estimation equations (7.87) 
and (7.88) where q; is defined by (7.89). 


(xx) In the evidence framework for RVM regression, we obtained the re-estimation 
formulae (7.87) and (7.88) by maximizing the marginal likelihood given by (7.85). 
Extend this approach by inclusion of hyperpriors given by gamma distributions of 
the form (B.26) and obtain the corresponding re-estimation formulae for œ and (3 by 
maximizing the corresponding posterior probability p(t, a, 3|X) with respect to a 
and (3. 


(xx) Derive the result (7.90) for the predictive distribution in the relevance vector 
machine for regression. Show that the predictive variance is given by (7.91). 


(« x) www | Using the results (7.94) and (7.95), show that the marginal likelihood 
(7.85) can be written in the form (7.96), where A(a,,) is defined by (7.97) and the 
sparsity and quality factors are defined by (7.98) and (7.99), respectively. 


(x) By taking the second derivative of the log marginal likelihood (7.97) for the 
regression RVM with respect to the hyperparameter a;, show that the stationary 
point given by (7.101) is a maximum of the marginal likelihood. 


(xx) Using (7.83) and (7.86), together with the matrix identity (C.7), show that 
the quantities S,, and Q,, defined by (7.102) and (7.103) can be written in the form 
(7.106) and (7.107). 


(x) PQQ Show that the gradient vector and Hessian matrix of the log poste- 
rior distribution (7.109) for the classification relevance vector machine are given by 
(7.110) and (7.111). 


(xx) Verify that maximization of the approximate log marginal likelihood function 
(7.114) for the classification relevance vector machine leads to the result (7.116) for 
re-estimation of the hyperparameters. 
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Probabilities play a central role in modern pattern recognition. We have seen in 
Chapter 1 that probability theory can be expressed in terms of two simple equations 
corresponding to the sum rule and the product rule. All of the probabilistic infer- 
ence and learning manipulations discussed in this book, no matter how complex, 
amount to repeated application of these two equations. We could therefore proceed 
to formulate and solve complicated probabilistic models purely by algebraic ma- 
nipulation. However, we shall find it highly advantageous to augment the analysis 
using diagrammatic representations of probability distributions, called probabilistic 
graphical models. These offer several useful properties: 


1. They provide a simple way to visualize the structure of a probabilistic model 
and can be used to design and motivate new models. 


2. Insights into the properties of the model, including conditional independence 
properties, can be obtained by inspection of the graph. 
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8.1. 


3. Complex computations, required to perform inference and learning in sophis- 
ticated models, can be expressed in terms of graphical manipulations, in which 
underlying mathematical expressions are carried along implicitly. 


A graph comprises nodes (also called vertices) connected by links (also known 
as edges or arcs). In a probabilistic graphical model, each node represents a random 
variable (or group of random variables), and the links express probabilistic relation- 
ships between these variables. The graph then captures the way in which the joint 
distribution over all of the random variables can be decomposed into a product of 
factors each depending only on a subset of the variables. We shall begin by dis- 
cussing Bayesian networks, also known as directed graphical models, in which the 
links of the graphs have a particular directionality indicated by arrows. The other 
major class of graphical models are Markov random fields, also known as undirected 
graphical models, in which the links do not carry arrows and have no directional 
significance. Directed graphs are useful for expressing causal relationships between 
random variables, whereas undirected graphs are better suited to expressing soft con- 
straints between random variables. For the purposes of solving inference problems, 
it is often convenient to convert both directed and undirected graphs into a different 
representation called a factor graph. 

In this chapter, we shall focus on the key aspects of graphical models as needed 
for applications in pattern recognition and machine learning. More general treat- 
ments of graphical models can be found in the books by Whittaker (1990), Lauritzen 
(1996), Jensen (1996), Castillo et al. (1997), Jordan (1999), Cowell et al. (1999), 
and Jordan (2007). 


Bayesian Networks 


In order to motivate the use of directed graphs to describe probability distributions, 
consider first an arbitrary joint distribution p(a, b, c) over three variables a, b, and c. 
Note that at this stage, we do not need to specify anything further about these vari- 
ables, such as whether they are discrete or continuous. Indeed, one of the powerful 
aspects of graphical models is that a specific graph can make probabilistic statements 
for a broad class of distributions. By application of the product rule of probability 
(1.11), we can write the joint distribution in the form 


p(a, b,c) = p(cla, b)p(a, b). (8.1) 


A second application of the product rule, this time to the second term on the right- 
hand side of (8.1), gives 


pla, b, c) = p(cla, b)p(bla)p(a). (8.2) 


Note that this decomposition holds for any choice of the joint distribution. We now 
represent the right-hand side of (8.2) in terms of a simple graphical model as follows. 
First we introduce a node for each of the random variables a, b, and c and associate 
each node with the corresponding conditional distribution on the right-hand side of 
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Figure 8.1 A directed graphical model representing the joint probabil- 4 
ity distribution over three variables a, b, and c, correspond- 
ing to the decomposition on the right-hand side of (8.2). 


oa 


(8.2). Then, for each conditional distribution we add directed links (arrows) to the 
graph from the nodes corresponding to the variables on which the distribution is 
conditioned. Thus for the factor p(c|a, b), there will be links from nodes a and b to 
node c, whereas for the factor p(a) there will be no incoming links. The result is the 
graph shown in Figure 8.1. If there is a link going from a node a to a node b, then we 
say that node a is the parent of node b, and we say that node b is the child of node a. 
Note that we shall not make any formal distinction between a node and the variable 
to which it corresponds but will simply use the same symbol to refer to both. 

An interesting point to note about (8.2) is that the left-hand side is symmetrical 
with respect to the three variables a, b, and c, whereas the right-hand side is not. 
Indeed, in making the decomposition in (8.2), we have implicitly chosen a particular 
ordering, namely a,b,c, and had we chosen a different ordering we would have 
obtained a different decomposition and hence a different graphical representation. 
We shall return to this point later. 

For the moment let us extend the example of Figure 8.1 by considering the joint 
distribution over K variables given by p(a1,...,2%«). By repeated application of 
the product rule of probability, this joint distribution can be written as a product of 
conditional distributions, one for each of the variables 


P Myon 5 OK) = PK |Cage s+ B21) +». plx2|xı)p(z1). (8.3) 


For a given choice of K, we can again represent this as a directed graph having K 
nodes, one for each conditional distribution on the right-hand side of (8.3), with each 
node having incoming links from all lower numbered nodes. We say that this graph 
is fully connected because there is a link between every pair of nodes. 

So far, we have worked with completely general joint distributions, so that the 
decompositions, and their representations as fully connected graphs, will be applica- 
ble to any choice of distribution. As we shall see shortly, it is the absence of links 
in the graph that conveys interesting information about the properties of the class of 
distributions that the graph represents. Consider the graph shown in Figure 8.2. This 
is not a fully connected graph because, for instance, there is no link from x to x2 or 
from x3 to x7. 

We shall now go from this graph to the corresponding representation of the joint 
probability distribution written in terms of the product of a set of conditional dis- 
tributions, one for each node in the graph. Each such conditional distribution will 
be conditioned only on the parents of the corresponding node in the graph. For in- 
stance, x5 will be conditioned on x, and x3. The joint distribution of all 7 variables 
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Figure 8.2 Example of a directed acyclic graph describing the joint 
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Exercise 8.2 


distribution over variables x1,...,27. The corresponding 
decomposition of the joint distribution is given by (8.4). 


is therefore given by 


p(z1)p(£2)p(£3)p(£4|x£1, £2, £3)p(x5|£1, 3) p(X6|L4)p(@7|@4, £5). (8.4) 


The reader should take a moment to study carefully the correspondence between 
(8.4) and Figure 8.2. 

We can now state in general terms the relationship between a given directed 
graph and the corresponding distribution over the variables. The joint distribution 
defined by a graph is given by the product, over all of the nodes of the graph, of 
a conditional distribution for each node conditioned on the variables corresponding 
to the parents of that node in the graph. Thus, for a graph with K nodes, the joint 
distribution is given by 


K 
p(x) = | [ perpar) (8.5) 
k=1 
where pa, denotes the set of parents of £k, and x = {21,...,vxK}. This key 


equation expresses the factorization properties of the joint distribution for a directed 
graphical model. Although we have considered each node to correspond to a single 
variable, we can equally well associate sets of variables and vector-valued variables 
with the nodes of a graph. It is easy to show that the representation on the right- 
hand side of (8.5) is always correctly normalized provided the individual conditional 
distributions are normalized. 

The directed graphs that we are considering are subject to an important restric- 
tion namely that there must be no directed cycles, in other words there are no closed 
paths within the graph such that we can move from node to node along links follow- 
ing the direction of the arrows and end up back at the starting node. Such graphs are 
also called directed acyclic graphs, or DAGs. This is equivalent to the statement that 
there exists an ordering of the nodes such that there are no links that go from any 
node to any lower numbered node. 


8.1.1 Example: Polynomial regression 


As an illustration of the use of directed graphs to describe probability distri- 
butions, we consider the Bayesian polynomial regression model introduced in Sec- 


Figure 8.3 


Figure 8.4 
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Directed graphical model representing the joint 
distribution (8.6) corresponding to the Bayesian 
polynomial regression model introduced in Sec- 
tion 1.2.6. 


tion 1.2.6. The random variables in this model are the vector of polynomial coeffi- 
cients w and the observed data t = (t1,...,¢ N)": In addition, this model contains 
the input data X = (x1,..., £ N)”, the noise variance o”, and the hyperparameter a 
representing the precision of the Gaussian prior over w, all of which are parameters 
of the model rather than random variables. Focussing just on the random variables 
for the moment, we see that the joint distribution is given by the product of the prior 
p(w) and N conditional distributions p(t, |w) for n = 1,...,.N so that 


N 
p(t, w “x tn|w). (8.6) 


This joint distribution can be represented by a graphical model shown in Figure 8.3. 


When we start to deal with more complex models later in the book, we shall find 
it inconvenient to have to write out multiple nodes of the form t1, . . . , ty explicitly as 
in Figure 8.3. We therefore introduce a graphical notation that allows such multiple 
nodes to be expressed more compactly, in which we draw a single representative 
node ¢,, and then surround this with a box, called a plate, labelled with N indicating 
that there are N nodes of this kind. Re-writing the graph of Figure 8.3 in this way, 
we obtain the graph shown in Figure 8.4. 

We shall sometimes find it helpful to make the parameters of a model, as well as 
its stochastic variables, explicit. In this case, (8.6) becomes 


p(t, w|X, a, o°) = p(wla) Tite |w, tn, 07). 


n= 


Correspondingly, we can make X and a explicit in the graphical representation. To 
do this, we shall adopt the convention that random variables will be denoted by open 
circles, and deterministic parameters will be denoted by smaller solid circles. If we 
take the graph of Figure 8.4 and include the deterministic parameters, we obtain the 
graph shown in Figure 8.5. 

When we apply a graphical model to a problem in machine learning or pattern 
recognition, we will typically set some of the random variables to specific observed 


An alternative, more compact, representation of the graph w 
shown in Figure 8.3 in which we have introduced a plate 

(the box labelled N) that represents N nodes of which only 

a single example tn is shown explicitly. 
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Figure 8.5 


Figure 8.6 


This shows the same model as in Figure 8.4 but 
with the deterministic parameters shown explicitly 
by the smaller solid nodes. 


values, for example the variables {tn } from the training set in the case of polynomial 
curve fitting. In a graphical model, we will denote such observed variables by shad- 
ing the corresponding nodes. Thus the graph corresponding to Figure 8.5 in which 
the variables {tn } are observed is shown in Figure 8.6. Note that the value of w is 
not observed, and so w is an example of a latent variable, also known as a hidden 
variable. Such variables play a crucial role in many probabilistic models and will 
form the focus of Chapters 9 and 12. 

Having observed the values {t,,} we can, if desired, evaluate the posterior dis- 
tribution of the polynomial coefficients w as discussed in Section 1.2.5. For the 
moment, we note that this involves a straightforward application of Bayes’ theorem 


p(w|T) x p(w TIn (ta|w) (8.7) 


n=1 


where again we have omitted the deterministic parameters in order to keep the nota- 
tion uncluttered. 

In general, model parameters such as w are of little direct interest in themselves, 
because our ultimate goal is to make predictions for new input values. Suppose we 
are given a new input value 7 and we wish to find the corresponding probability dis- 
tribution for t conditioned on the observed data. The graphical model that describes 
this problem is shown in Figure 8.7, and the corresponding joint distribution of all 
of the random variables in this model, conditioned on the deterministic parameters, 
is then given by 


N 
p(t, t, w|Z,X, a,07) = TI Helen?) p(wla)p(t]Z, w, o°). (8.8) 
n=1 


As in Figure 8.5 but with the nodes {t,,} shaded 
to indicate that the corresponding random vari- 
ables have been set to their observed (training set) 
values. 
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Figure 8.7 The polynomial regression model, corresponding 
to Figure 8.6, showing also a new input value © 
together with the corresponding model prediction 
t. 


The required predictive distribution for T is then obtained, from the sum rule of 
probability, by integrating out the model parameters w so that 


PAZ, x,t, a, 0?) « [ r@twle,x.0,0%) aw 


where we are implicitly setting the random variables in t to the specific values ob- 
served in the data set. The details of this calculation were discussed in Chapter 3. 


8.1.2 Generative models 


There are many situations in which we wish to draw samples from a given prob- 
ability distribution. Although we shall devote the whole of Chapter 11 to a detailed 
discussion of sampling methods, it is instructive to outline here one technique, called 
ancestral sampling, which is particularly relevant to graphical models. Consider a 
joint distribution p(x1,..., £g) over K variables that factorizes according to (8.5) 
corresponding to a directed acyclic graph. We shall suppose that the variables have 
been ordered such that there are no links from any node to any lower numbered node, 
in other words each node has a higher number than any of its parents. Our goal is to 
draw a sample 7,..., from the joint distribution. 

To do this, we start with the lowest-numbered node and draw a sample from the 
distribution p(21), which we call £1. We then work through each of the nodes in or- 
der, so that for node n we draw a sample from the conditional distribution p(x,,|pa,, ) 
in which the parent variables have been set to their sampled values. Note that at each 
stage, these parent values will always be available because they correspond to lower- 
numbered nodes that have already been sampled. Techniques for sampling from 
specific distributions will be discussed in detail in Chapter 11. Once we have sam- 
pled from the final variable xx, we will have achieved our objective of obtaining a 
sample from the joint distribution. To obtain a sample from some marginal distribu- 
tion corresponding to a subset of the variables, we simply take the sampled values 
for the required nodes and ignore the sampled values for the remaining nodes. For 
example, to draw a sample from the distribution p(x2, 24), we simply sample from 
the full joint distribution and then retain the values 72, 74 and discard the remaining 
values {2 j42,4}. 
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Figure 8.8 A graphical model representing the process by which Object Position Orientation 


Section 2.4 


images of objects are created, in which the identity 
of an object (a discrete variable) and the position and 
orientation of that object (continuous variables) have 
independent prior probabilities. The image (a vector 
of pixel intensities) has a probability distribution that 
is dependent on the identity of the object as well as 
on its position and orientation. 


Image 


For practical applications of probabilistic models, it will typically be the higher- 
numbered variables corresponding to terminal nodes of the graph that represent the 
observations, with lower-numbered nodes corresponding to latent variables. The 
primary role of the latent variables is to allow a complicated distribution over the 
observed variables to be represented in terms of a model constructed from simpler 
(typically exponential family) conditional distributions. 

We can interpret such models as expressing the processes by which the observed 
data arose. For instance, consider an object recognition task in which each observed 
data point corresponds to an image (comprising a vector of pixel intensities) of one 
of the objects. In this case, the latent variables might have an interpretation as the 
position and orientation of the object. Given a particular observed image, our goal is 
to find the posterior distribution over objects, in which we integrate over all possible 
positions and orientations. We can represent this problem using a graphical model 
of the form show in Figure 8.8. 

The graphical model captures the causal process (Pearl, 1988) by which the ob- 
served data was generated. For this reason, such models are often called generative 
models. By contrast, the polynomial regression model described by Figure 8.5 is 
not generative because there is no probability distribution associated with the input 
variable x, and so it is not possible to generate synthetic data points from this model. 
We could make it generative by introducing a suitable prior distribution p(x), at the 
expense of a more complex model. 

The hidden variables in a probabilistic model need not, however, have any ex- 
plicit physical interpretation but may be introduced simply to allow a more complex 
joint distribution to be constructed from simpler components. In either case, the 
technique of ancestral sampling applied to a generative model mimics the creation 
of the observed data and would therefore give rise to ‘fantasy’ data whose probability 
distribution (if the model were a perfect representation of reality) would be the same 
as that of the observed data. In practice, producing synthetic observations from a 
generative model can prove informative in understanding the form of the probability 
distribution represented by that model. 


8.1.3 Discrete variables 


We have discussed the importance of probability distributions that are members 
of the exponential family, and we have seen that this family includes many well- 
known distributions as particular cases. Although such distributions are relatively 
simple, they form useful building blocks for constructing more complex probability 


Figure 8.9 
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(a) This fully-connected graph describes a general distribu- Xı X2 


tion over two K-state discrete variables having a total of a) O— O 
K? — 1 parameters. (b) By dropping the link between the 
nodes, the number of parameters is reduced to 2(K — 1). xy Xo 


distributions, and the framework of graphical models is very useful in expressing the 
way in which these building blocks are linked together. 

Such models have particularly nice properties if we choose the relationship be- 
tween each parent-child pair in a directed graph to be conjugate, and we shall ex- 
plore several examples of this shortly. Two cases are particularly worthy of note, 
namely when the parent and child node each correspond to discrete variables and 
when they each correspond to Gaussian variables, because in these two cases the 
relationship can be extended hierarchically to construct arbitrarily complex directed 
acyclic graphs. We begin by examining the discrete case. 

The probability distribution p(x|jz) for a single discrete variable x having K 
possible states (using the 1-of-/ representation) is given by 


p(x|u) = Nes Me (8.9) 


and is governed by the parameters u = (41,..-, ug)". Due to the constraint 
X; Hk = 1, only K — 1 values for py need to be specified in order to define the 
distribution. 

Now suppose that we have two discrete variables, x, and x2, each of which has 
K states, and we wish to model their joint distribution. We denote the probability of 
observing both 71, = 1 and x2; = 1 by the parameter upi, where xı% denotes the 
kt? component of x,, and similarly for 22;. The joint distribution can be written 


p(X1, X2|u) = Tle 


k=1 l=1 


Because the parameters up, are subject to the constraint 5 k 5 ı kl = 1, this distri- 
bution is governed by K? — 1 parameters. It is easily seen that the total number of 
parameters that must be specified for an arbitrary joint distribution over M variables 
is K™ — 1 and therefore grows exponentially with the number M of variables. 

Using the product rule, we can factor the joint distribution p(x, x2) in the form 
p(X2|X1)p(x1), which corresponds to a two-node graph with a link going from the 
xı node to the x2 node as shown in Figure 8.9(a). The marginal distribution p(x;) 
is governed by K — 1 parameters, as before, Similarly, the conditional distribution 
p(X2|X1) requires the specification of K — 1 parameters for each of the K possible 
values of xı. The total number of parameters that must be specified in the joint 
distribution is therefore (K — 1) + K(K — 1) = K? — 1 as before. 

Now suppose that the variables x; and x2 were independent, corresponding to 
the graphical model shown in Figure 8.9(b). Each variable is then described by 
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Figure 8.10 This chain of M discrete nodes, each *1 X2 XM 
having K states, requires the specification of K — 1 + O—QO— ET —() 
(M — 1)K(K — 1) parameters, which grows linearly 

with the length M of the chain. In contrast, a fully con- 


nected graph of M nodes would have K™ — 1 param- 
eters, which grows exponentially with M. 


a separate multinomial distribution, and the total number of parameters would be 
2(K — 1). For a distribution over M independent discrete variables, each having K 
states, the total number of parameters would be M (K — 1), which therefore grows 
linearly with the number of variables. From a graphical perspective, we have reduced 
the number of parameters by dropping links in the graph, at the expense of having a 
restricted class of distributions. 

More generally, if we have M discrete variables x;,...,x),z, we can model 
the joint distribution using a directed graph with one variable corresponding to each 
node. The conditional distribution at each node is given by a set of nonnegative pa- 
rameters subject to the usual normalization constraint. If the graph is fully connected 
then we have a completely general distribution having K™ — 1 parameters, whereas 
if there are no links in the graph the joint distribution factorizes into the product of 
the marginals, and the total number of parameters is M(K — 1). Graphs having in- 
termediate levels of connectivity allow for more general distributions than the fully 
factorized one while requiring fewer parameters than the general joint distribution. 
As an illustration, consider the chain of nodes shown in Figure 8.10. The marginal 
distribution p(x,) requires K — 1 parameters, whereas each of the M — 1 condi- 
tional distributions p(x;|xX;—1), fori = 2,..., M, requires K(K — 1) parameters. 
This gives a total parameter count of K — 1+ (M —1)K(K — 1), which is quadratic 
in K and which grows linearly (rather than exponentially) with the length M of the 
chain. 

An alternative way to reduce the number of independent parameters in a model 
is by sharing parameters (also known as tying of parameters). For instance, in the 
chain example of Figure 8.10, we can arrange that all of the conditional distributions 
p(xi|xi—1), for i = 2,..., M, are governed by the same set of K(K—1) parameters. 
Together with the K —1 parameters governing the distribution of xj, this gives a total 
of K? — 1 parameters that must be specified in order to define the joint distribution. 

We can turn a graph over discrete variables into a Bayesian model by introduc- 
ing Dirichlet priors for the parameters. From a graphical point of view, each node 
then acquires an additional parent representing the Dirichlet distribution over the pa- 
rameters associated with the corresponding discrete node. This is illustrated for the 
chain model in Figure 8.11. The corresponding model in which we tie the parame- 
ters governing the conditional distributions p(x;|x;_1), for i = 2,..., M, is shown 
in Figure 8.12. 

Another way of controlling the exponential growth in the number of parameters 
in models of discrete variables is to use parameterized models for the conditional 
distributions instead of complete tables of conditional probability values. To illus- 
trate this idea, consider the graph in Figure 8.13 in which all of the nodes represent 
binary variables. Each of the parent variables x; is governed by a single parame- 


Figure 8.11 


Figure 8.12 
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Figure 8.13 
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An extension of the model of Hı H2 HM 
Figure 8.10 to include Dirich- 

let priors over the param- 

eters governing the discrete 

distributions. 


X1 X2 XM 


As in Figure 8.11 but with a sin- 
gle set of parameters u shared 
amongst all of the conditional 
distributions p(x;|x:-1). 


ter u; representing the probability p(x; = 1), giving M parameters in total for the 
parent nodes. The conditional distribution p(y|x1,..., 211), however, would require 
2™ parameters representing the probability p(y = 1) for each of the 2™ possible 
settings of the parent variables. Thus in general the number of parameters required 
to specify this conditional distribution will grow exponentially with W. We can ob- 
tain a more parsimonious form for the conditional distribution by using a logistic 
sigmoid function acting on a linear combination of the parent variables, giving 


M 
ply = 1|z1,... £M) =o TEDD = o(w'x) (8.10) 


i=1 


where o(a) = (1+exp(—a))~? is the logistic sigmoid, x = (xo, £1,- .. , £m)” isan 
(M + 1)-dimensional vector of parent states augmented with an additional variable 
xo whose value is clamped to 1, and w = (wo, W1,---, wm)” is a vector of M +1 
parameters. This is a more restricted form of conditional distribution than the general 
case but is now governed by a number of parameters that grows linearly with M. In 
this sense, it is analogous to the choice of a restrictive form of covariance matrix (for 
example, a diagonal matrix) in a multivariate Gaussian distribution. The motivation 
for the logistic sigmoid representation was discussed in Section 4.2. 


A graph comprising M parents zx1,..., xm and a sin- Tı TM 
gle child y, used to illustrate the idea of parameterized 
conditional distributions for discrete variables. 
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8.1.4 Linear-Gaussian models 


In the previous section, we saw how to construct joint probability distributions 
over a set of discrete variables by expressing the variables as nodes in a directed 
acyclic graph. Here we show how a multivariate Gaussian can be expressed as a 
directed graph corresponding to a linear-Gaussian model over the component vari- 
ables. This allows us to impose interesting structure on the distribution, with the 
general Gaussian and the diagonal covariance Gaussian representing opposite ex- 
tremes. Several widely used techniques are examples of linear-Gaussian models, 
such as probabilistic principal component analysis, factor analysis, and linear dy- 
namical systems (Roweis and Ghahramani, 1999). We shall make extensive use of 
the results of this section in later chapters when we consider some of these techniques 
in detail. 

Consider an arbitrary directed acyclic graph over D variables in which node 2 
represents a single continuous random variable x; having a Gaussian distribution. 
The mean of this distribution is taken to be a linear combination of the states of its 
parent nodes pa, of node 7 


p(x;|pa;) =N | zi >, Wig Lz + bi, UV; (8.11) 
jEpa; 
where wij and b; are parameters governing the mean, and v; is the variance of the 


conditional distribution for x;. The log of the joint distribution is then the log of the 
product of these conditionals over all nodes in the graph and hence takes the form 


D 
Inp(x) = S-Inp(a;|pa,) (8.12) 
i=1 
2 
ae 
= 2v; Ti > Wijtj — bi + const (8.13) 
i=1 jEpa; 
where x = (x1,...,£p)" and ‘const’ denotes terms independent of x. We see that 


this is a quadratic function of the components of x, and hence the joint distribution 
p(x) is a multivariate Gaussian. 

We can determine the mean and covariance of the joint distribution recursively 
as follows. Each variable x; has (conditional on the states of its parents) a Gaussian 
distribution of the form (8.11) and so 


Ti = > Wijk 5 + b; + vici (8.14) 


jEpa; 


where €; is a zero mean, unit variance Gaussian random variable satisfying E[e;] = 0 
and Ele;e;] = Tij, where I;; is the i, j element of the identity matrix. Taking the 
expectation of (8.14), we have 


Efx] = 5 Wij E(x 5] + bi. (8.15) 
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Figure 8.14 A directed graph over three Gaussian variables, Tı T2 T3 
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Exercise 8.7 


with one missing link. O—O— O 


Thus we can find the components of Efx] = (E[xı], ..., Eļ£p])T by starting at the 
lowest numbered node and working recursively through the graph (here we again 
assume that the nodes are numbered such that each node has a higher number than 
its parents). Similarly, we can use (8.14) and (8.15) to obtain the 7, j element of the 
covariance matrix for p(x) in the form of a recursion relation 


cov[xi,2;] = E[(x;— E[x,])(x; — Elz;])] 


= E | (a; — E[z;]) `> Wyk (Le — Elag]) + 056; 


= iA wjpcov[zi, Tg] + Iijvj (8.16) 


ke paj 


and so the covariance can similarly be evaluated recursively starting from the lowest 
numbered node. 

Let us consider two extreme cases. First of all, suppose that there are no links 
in the graph, which therefore comprises D isolated nodes. In this case, there are no 
parameters w;j and so there are just D parameters b; and D parameters v;. From 
the recursion relations (8.15) and (8.16), we see that the mean of p(x) is given by 
(bi,...,bp)* and the covariance matrix is diagonal of the form diag(v1,..., vp). 
The joint distribution has a total of 2D parameters and represents a set of D inde- 
pendent univariate Gaussian distributions. 

Now consider a fully connected graph in which each node has all lower num- 
bered nodes as parents. The matrix w;j then has i — 1 entries on the i‘ row and 
hence is a lower triangular matrix (with no entries on the leading diagonal). Then 
the total number of parameters w;; is obtained by taking the number D? of elements 
ina D x D matrix, subtracting D to account for the absence of elements on the lead- 
ing diagonal, and then dividing by 2 because the matrix has elements only below the 
diagonal, giving a total of D( D —1)/2. The total number of independent parameters 
{wij} and {v;} in the covariance matrix is therefore D(D + 1)/2 corresponding to 
a general symmetric covariance matrix. 

Graphs having some intermediate level of complexity correspond to joint Gaus- 
sian distributions with partially constrained covariance matrices. Consider for ex- 
ample the graph shown in Figure 8.14, which has a link missing between variables 
x, and x3. Using the recursion relations (8.15) and (8.16), we see that the mean and 
covariance of the joint distribution are given by 


u = (b1, bg + w21b1, b3 + Ww32b2 + W32W21b1)" (8.17) 
Ui W211 W32W21 U1 
> = W21U1 U2 + wv w32(Ve2 + w201) 7 (8.18) 
W32W21V1 W32(V2 + W301) U3 + W3o(U2 ae w301) 
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We can readily extend the linear-Gaussian graphical model to the case in which 
the nodes of the graph represent multivariate Gaussian variables. In this case, we can 
write the conditional distribution for node 7 in the form 


p(x;|pa;) =N | x: 5 Wx; + bi, Xi (8.19) 


jEpa; 


where now W;; is a matrix (which is nonsquare if x; and x; have different dimen- 
sionalities). Again it is easy to verify that the joint distribution over all variables is 
Gaussian. 

Note that we have already encountered a specific example of the linear-Gaussian 
relationship when we saw that the conjugate prior for the mean yz of a Gaussian 
variable x is itself a Gaussian distribution over yz. The joint distribution over x and 
pt is therefore Gaussian. This corresponds to a simple two-node graph in which 
the node representing p is the parent of the node representing x. The mean of the 
distribution over jz is a parameter controlling a prior, and so it can be viewed as a 
hyperparameter. Because the value of this hyperparameter may itself be unknown, 
we can again treat it from a Bayesian perspective by introducing a prior over the 
hyperparameter, sometimes called a hyperprior, which is again given by a Gaussian 
distribution. This type of construction can be extended in principle to any level and is 
an illustration of a hierarchical Bayesian model, of which we shall encounter further 
examples in later chapters. 


Conditional Independence 


An important concept for probability distributions over multiple variables is that of 
conditional independence (Dawid, 1980). Consider three variables a, b, and c, and 
suppose that the conditional distribution of a, given b and c, is such that it does not 
depend on the value of b, so that 


p(alb, c) = p(alc). (8.20) 


We say that a is conditionally independent of b given c. This can be expressed in a 
slightly different way if we consider the joint distribution of a and b conditioned on 
c, which we can write in the form 


p(a, ble) = p(alb,c)p(d|c) 
= plale)p(b|c). (8.21) 


where we have used the product rule of probability together with (8.20). Thus we 
see that, conditioned on c, the joint distribution of a and b factorizes into the prod- 
uct of the marginal distribution of a and the marginal distribution of b (again both 
conditioned on c). This says that the variables a and b are statistically independent, 
given c. Note that our definition of conditional independence will require that (8.20), 
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Figure 8.15 The first of three examples of graphs over three variables C 
a, b, and c used to discuss conditional independence 
properties of directed graphical models. 


a 
Er 


or equivalently (8.21), must hold for every possible value of c, and not just for some 
values. We shall sometimes use a shorthand notation for conditional independence 
(Dawid, 1979) in which 

allbj|c (8.22) 


denotes that a is conditionally independent of b given c and is equivalent to (8.20). 

Conditional independence properties play an important role in using probabilis- 
tic models for pattern recognition by simplifying both the structure of a model and 
the computations needed to perform inference and learning under that model. We 
shall see examples of this shortly. 

If we are given an expression for the joint distribution over a set of variables in 
terms of a product of conditional distributions (i.e., the mathematical representation 
underlying a directed graph), then we could in principle test whether any poten- 
tial conditional independence property holds by repeated application of the sum and 
product rules of probability. In practice, such an approach would be very time con- 
suming. An important and elegant feature of graphical models is that conditional 
independence properties of the joint distribution can be read directly from the graph 
without having to perform any analytical manipulations. The general framework 
for achieving this is called d-separation, where the ‘d’ stands for ‘directed’ (Pearl, 
1988). Here we shall motivate the concept of d-separation and give a general state- 
ment of the d-separation criterion. A formal proof can be found in Lauritzen (1996). 


8.2.1 Three example graphs 


We begin our discussion of the conditional independence properties of directed 
graphs by considering three simple examples each involving graphs having just three 
nodes. Together, these will motivate and illustrate the key concepts of d-separation. 
The first of the three examples is shown in Figure 8.15, and the joint distribution 
corresponding to this graph is easily written down using the general result (8.5) to 
give 

pla, b,c) = p(ale)p(b|e)p(e). (8.23) 
If none of the variables are observed, then we can investigate whether a and b are 
independent by marginalizing both sides of (8.23) with respect to c to give 


p(a, b) = X` p(ale)p(ble)p(c). (8.24) 


In general, this does not factorize into the product p(a)p(b), and so 


all bjó (8.25) 
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Figure 8.16 


Figure 8.17 


As in Figure 8.15 but where we have conditioned on the c 
value of variable c. 


Q 
e 


where Ø denotes the empty set, and the symbol -X means that the conditional inde- 
pendence property does not hold in general. Of course, it may hold for a particular 
distribution by virtue of the specific numerical values associated with the various 
conditional probabilities, but it does not follow in general from the structure of the 
graph. 

Now suppose we condition on the variable c, as represented by the graph of 
Figure 8.16. From (8.23), we can easily write down the conditional distribution of a 
and b, given c, in the form 


pla, b,c) 
p(c) 
= p(alc)p(blc) 


pla, bc) 


and so we obtain the conditional independence property 
a ib |e, 


We can provide a simple graphical interpretation of this result by considering 
the path from node a to node b via c. The node c is said to be tail-to-tail with re- 
spect to this path because the node is connected to the tails of the two arrows, and 
the presence of such a path connecting nodes a and b causes these nodes to be de- 
pendent. However, when we condition on node c, as in Figure 8.16, the conditioned 
node ‘blocks’ the path from a to b and causes a and b to become (conditionally) 
independent. 

We can similarly consider the graph shown in Figure 8.17. The joint distribution 
corresponding to this graph is again obtained from our general formula (8.5) to give 


p(a, b,c) = p(a)p(cla)p(d|c). (8.26) 


First of all, suppose that none of the variables are observed. Again, we can test to 
see if a and b are independent by marginalizing over c to give 


p(a,b) = p(a) X p(cla)p(b|e) = p(a)p(bla). 


The second of our three examples of 3-node a C b 


graphs used to motivate the conditional indepen- ( ) > ) > ) 
dence framework for directed graphical models. 
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Figure 8.18 As in Figure 8.17 but now conditioning on node c. a c b 


Figure 8.19 


O—@—O 


which in general does not factorize into p(a)p(b), and so 
ablo (8.27) 


as before. 
Now suppose we condition on node c, as shown in Figure 8.18. Using Bayes’ 
theorem, together with (8.26), we obtain 


p(a, ble) 


= plale)p(e|c) 
and so again we obtain the conditional independence property 
a tl b|c. 


As before, we can interpret these results graphically. The node c is said to be 
head-to-tail with respect to the path from node a to node b. Such a path connects 
nodes a and b and renders them dependent. If we now observe c, as in Figure 8.18, 
then this observation ‘blocks’ the path from a to b and so we obtain the conditional 
independence property a LL b | c. 

Finally, we consider the third of our 3-node examples, shown by the graph in 
Figure 8.19. As we shall see, this has a more subtle behaviour than the two previous 
graphs. 

The joint distribution can again be written down using our general result (8.5) to 
give 

p(a, b,c) = p(a)p(b)p(cla, b). (8.28) 


Consider first the case where none of the variables are observed. Marginalizing both 
sides of (8.28) over c we obtain 


p(a, b) = p(a)p(b) 


The last of our three examples of 3-node graphs used to a b 
explore conditional independence properties in graphi- 

cal models. This graph has rather different properties 

from the two previous examples. 
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Figure 8.20 As in Figure 8.19 but conditioning on the value of node a b 


Exercise 8.10 


c. In this graph, the act of conditioning induces a depen- 
dence between a and b. 


and so a and b are independent with no variables observed, in contrast to the two 
previous examples. We can write this result as 


all b|0. (8.29) 


Now suppose we condition on c, as indicated in Figure 8.20. The conditional distri- 
bution of a and b is then given by 


p(a, b,c) 
p(c) 
p(a)p(b)p(cla, b) 


p(c) 


pla, bļe) 


which in general does not factorize into the product p(a)p(b), and so 


ay bl c. 


Thus our third example has the opposite behaviour from the first two. Graphically, 
we say that node c is head-to-head with respect to the path from a to b because it 
connects to the heads of the two arrows. When node c is unobserved, it ‘blocks’ 
the path, and the variables a and b are independent. However, conditioning on c 
‘unblocks’ the path and renders a and b dependent. 

There is one more subtlety associated with this third example that we need to 
consider. First we introduce some more terminology. We say that node y is a de- 
scendant of node «x if there is a path from x to y in which each step of the path 
follows the directions of the arrows. Then it can be shown that a head-to-head path 
will become unblocked if either the node, or any of its descendants, is observed. 

In summary, a tail-to-tail node or a head-to-tail node leaves a path unblocked 
unless it is observed in which case it blocks the path. By contrast, a head-to-head 
node blocks a path if it is unobserved, but once the node, and/or at least one of its 
descendants, is observed the path becomes unblocked. 

It is worth spending a moment to understand further the unusual behaviour of the 
graph of Figure 8.20. Consider a particular instance of such a graph corresponding 
to a problem with three binary random variables relating to the fuel system on a car, 
as shown in Figure 8.21. The variables are called B, representing the state of a 
battery that is either charged (B = 1) or flat (B = 0), F representing the state of 
the fuel tank that is either full of fuel (F = 1) or empty (F = 0), and G, which is 
the state of an electric fuel gauge and which indicates either full (G = 1) or empty 
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B F B F B F 


G G G 


Figure 8.21 An example of a 3-node graph used to illustrate the phenomenon of ‘explaining away’. The three 
nodes represent the state of the battery (B), the state of the fuel tank (F) and the reading on the electric fuel 
gauge (G). See the text for details. 


(G = 0). The battery is either charged or flat, and independently the fuel tank is 


either full or empty, with prior probabilities 
1) = 09 
1) = 0.9. 


Given the state of the fuel tank and the battery, the fuel gauge reads full with proba- 
bilities given by 


p(G=1|B=1,F=1) 0.8 
p(G=1|B=1,F =0) 0.2 
p(G =1|B =0,F =1) 0.2 
p(G = 1|B =0,F =0) 0.1 


so this is a rather unreliable fuel gauge! All remaining probabilities are determined 
by the requirement that probabilities sum to one, and so we have a complete specifi- 
cation of the probabilistic model. 

Before we observe any data, the prior probability of the fuel tank being empty 
is p(F = 0) = 0.1. Now suppose that we observe the fuel gauge and discover that 
it reads empty, i.e., G = 0, corresponding to the middle graph in Figure 8.21. We 
can use Bayes’ theorem to evaluate the posterior probability of the fuel tank being 
empty. First we evaluate the denominator for Bayes’ theorem given by 


p(G=0)= X` YS) p(G=0B, F)p(B)p(F) = 0.315 (8.30) 
Be{0,1} F€{0,1} 
and similarly we evaluate 
p(G=0F=0)= $ p(G=0|B,F =0)p(B) =0.81 (8.31) 
Be{0,1} 


and using these results we have 


_ A(G = 0|F = 0)p(F = 0) 
p(G = 0) 


p(F =0|G =0) ~ 0.257 (8.32) 
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and so p(f = 0|G = 0) > p(F = 0). Thus observing that the gauge reads empty 
makes it more likely that the tank is indeed empty, as we would intuitively expect. 
Next suppose that we also check the state of the battery and find that it is flat, i.e., 
B = 0. We have now observed the states of both the fuel gauge and the battery, as 
shown by the right-hand graph in Figure 8.21. The posterior probability that the fuel 
tank is empty given the observations of both the fuel gauge and the battery state is 
then given by 


p(G = 0|B = 0, F =0)p(F =0) 
Do Fe{o,1} p(G = 0|B = 0, F)p(F) 


where the prior probability p(B = 0) has cancelled between numerator and denom- 
inator. Thus the probability that the tank is empty has decreased (from 0.257 to 
0.111) as a result of the observation of the state of the battery. This accords with our 
intuition that finding out that the battery is flat explains away the observation that the 
fuel gauge reads empty. We see that the state of the fuel tank and that of the battery 
have indeed become dependent on each other as a result of observing the reading 
on the fuel gauge. In fact, this would also be the case if, instead of observing the 
fuel gauge directly, we observed the state of some descendant of G. Note that the 
probability p(F = 0|G = 0, B = 0) ~ 0.111 is greater than the prior probability 
p(F = 0) = 0.1 because the observation that the fuel gauge reads zero still provides 
some evidence in favour of an empty fuel tank. 


p(F = 0|G = 0, B =0) ~ 0.111 (8.33) 


8.2.2 D-separation 


We now give a general statement of the d-separation property (Pearl, 1988) for 
directed graphs. Consider a general directed graph in which A, B, and C are arbi- 
trary nonintersecting sets of nodes (whose union may be smaller than the complete 
set of nodes in the graph). We wish to ascertain whether a particular conditional 
independence statement A 1L B | C is implied by a given directed acyclic graph. To 
do so, we consider all possible paths from any node in A to any node in B. Any such 
path is said to be blocked if it includes a node such that either 


(a) the arrows on the path meet either head-to-tail or tail-to-tail at the node, and the 
node is in the set C, or 


(b) the arrows meet head-to-head at the node, and neither the node, nor any of its 
descendants, is in the set C. 


If all paths are blocked, then A is said to be d-separated from B by C, and the joint 
distribution over all of the variables in the graph will satisfy A 1L B | C. 

The concept of d-separation is illustrated in Figure 8.22. In graph (a), the path 
from a to b is not blocked by node f because it is a tail-to-tail node for this path 
and is not observed, nor is it blocked by node e because, although the latter is a 
head-to-head node, it has a descendant c because is in the conditioning set. Thus 
the conditional independence statement a LL b | c does not follow from this graph. 
In graph (b), the path from a to b is blocked by node f because this is a tail-to-tail 
node that is observed, and so the conditional independence property a IL b | f will 
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Figure 8.22 Illustration of the con- gq f a f 
cept of d-separation. See the text for 


details. 


Section 2.3 


Figure 8.23 


(a) (b) 


be satisfied by any distribution that factorizes according to this graph. Note that this 
path is also blocked by node e because e is a head-to-head node and neither it nor its 
descendant are in the conditioning set. 

For the purposes of d-separation, parameters such as œ and o? in Figure 8.5, 
indicated by small filled circles, behave in the same was as observed nodes. How- 
ever, there are no marginal distributions associated with such nodes. Consequently 
parameter nodes never themselves have parents and so all paths through these nodes 
will always be tail-to-tail and hence blocked. Consequently they play no role in 
d-separation. 

Another example of conditional independence and d-separation is provided by 
the concept of i.i.d. (independent identically distributed) data introduced in Sec- 
tion 1.2.4. Consider the problem of finding the posterior distribution for the mean 
of a univariate Gaussian distribution. This can be represented by the directed graph 
shown in Figure 8.23 in which the joint distribution is defined by a prior p(s) to- 
gether with a set of conditional distributions p(z,,|) for n = 1,..., N. In practice, 
we observe D = {2 ,...,y} and our goal is to infer u. Suppose, for a moment, 
that we condition on u and consider the joint distribution of the observations. Using 
d-separation, we note that there is a unique path from any x; to any other x;z; and 
that this path is tail-to-tail with respect to the observed node u. Every such path is 


blocked and so the observations D = {x,,...,2, } are independent given u, so that 
N 
p(D\u) = | [ enle). (8.34) 
n=1 
(a) Directed graph corre- H 


sponding to the problem 
of inferring the mean p of 
a univariate Gaussian dis- 
tribution from observations 
z1ı,...,&ẸN. (b) The same 
graph drawn using the plate 
notation. 
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Figure 8.24 


Section 3.3 


A graphical representation of the ‘naive Bayes’ 
model for classification. | Conditioned on the 
class label z, the components of the observed 
vector x = (m1,...,2p)" are assumed to be 
independent. 


However, if we integrate over ju, the observations are in general no longer indepen- 
dent 


fore) N 
wD) = | ool dn + T] pen) (8.35) 


Here 4 is a latent variable, because its value is not observed. 

Another example of a model representing i.i.d. data is the graph in Figure 8.7 
corresponding to Bayesian polynomial regression. Here the stochastic nodes corre- 
spond to {tn}, w and T. We see that the node for w is tail-to-tail with respect to 
the path from £ to any one of the nodes t» and so we have the following conditional 
independence property 2 

t IL ty | w. (8.36) 


Thus, conditioned on the polynomial coefficients w, the predictive distribution for 
T is independent of the training data {ti,...,¢n}. We can therefore first use the 
training data to determine the posterior distribution over the coefficients w and then 
we can discard the training data and use the posterior distribution for w to make 
predictions of t for new input observations F. 

A related graphical structure arises in an approach to classification called the 
naive Bayes model, in which we use conditional independence assumptions to sim- 
plify the model structure. Suppose our observed variable consists of a D-dimensional 
vector x = (£1,... ,tp)' and we wish to assign observed values of x to one of K 
classes. Using the 1-of- encoding scheme, we can represent these classes by a K- 
dimensional binary vector z. We can then define a generative model by introducing 
a multinomial prior p(z|j) over the class labels, where the k** component ju, of p 
is the prior probability of class C, together with a conditional distribution p(x|z) 
for the observed vector x. The key assumption of the naive Bayes model is that, 
conditioned on the class z, the distributions of the input variables 71,..., £p are in- 
dependent. The graphical representation of this model is shown in Figure 8.24. We 
see that observation of z blocks the path between x; and x; for 7 4 i (because such 
paths are tail-to-tail at the node z) and so x; and x; are conditionally independent 
given z. If, however, we marginalize out z (so that z is unobserved) the tail-to-tail 
path from x; to x; is no longer blocked. This tells us that in general the marginal 
density p(x) will not factorize with respect to the components of x. We encountered 
a simple application of the naive Bayes model in the context of fusing data from 
different sources for medical diagnosis in Section 1.5. 

If we are given a labelled training set, comprising inputs {x,,..., Xy } together 
with their class labels, then we can fit the naive Bayes model to the training data 
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using maximum likelihood assuming that the data are drawn independently from 
the model. The solution is obtained by fitting the model for each class separately 
using the correspondingly labelled data. As an example, suppose that the probability 
density within each class is chosen to be Gaussian. In this case, the naive Bayes 
assumption then implies that the covariance matrix for each Gaussian is diagonal, 
and the contours of constant density within each class will be axis-aligned ellipsoids. 
The marginal density, however, is given by a superposition of diagonal Gaussians 
(with weighting coefficients given by the class priors) and so will no longer factorize 
with respect to its components. 

The naive Bayes assumption is helpful when the dimensionality D of the input 
space is high, making density estimation in the full D-dimensional space more chal- 
lenging. It is also useful if the input vector contains both discrete and continuous 
variables, since each can be represented separately using appropriate models (e.g., 
Bernoulli distributions for binary observations or Gaussians for real-valued vari- 
ables). The conditional independence assumption of this model is clearly a strong 
one that may lead to rather poor representations of the class-conditional densities. 
Nevertheless, even if this assumption is not precisely satisfied, the model may still 
give good classification performance in practice because the decision boundaries can 
be insensitive to some of the details in the class-conditional densities, as illustrated 
in Figure 1.27. 

We have seen that a particular directed graph represents a specific decomposition 
of a joint probability distribution into a product of conditional probabilities. The 
graph also expresses a set of conditional independence statements obtained through 
the d-separation criterion, and the d-separation theorem is really an expression of the 
equivalence of these two properties. In order to make this clear, it is helpful to think 
of a directed graph as a filter. Suppose we consider a particular joint probability 
distribution p(x) over the variables x corresponding to the (nonobserved) nodes of 
the graph. The filter will allow this distribution to pass through if, and only if, it can 
be expressed in terms of the factorization (8.5) implied by the graph. If we present to 
the filter the set of all possible distributions p(x) over the set of variables x, then the 
subset of distributions that are passed by the filter will be denoted DF, for directed 
factorization. This is illustrated in Figure 8.25. Alternatively, we can use the graph as 
a different kind of filter by first listing all of the conditional independence properties 
obtained by applying the d-separation criterion to the graph, and then allowing a 
distribution to pass only if it satisfies all of these properties. If we present all possible 
distributions p(x) to this second kind of filter, then the d-separation theorem tells us 
that the set of distributions that will be allowed through is precisely the set DF. 

It should be emphasized that the conditional independence properties obtained 
from d-separation apply to any probabilistic model described by that particular di- 
rected graph. This will be true, for instance, whether the variables are discrete or 
continuous or a combination of these. Again, we see that a particular graph is de- 
scribing a whole family of probability distributions. 

At one extreme we have a fully connected graph that exhibits no conditional in- 
dependence properties at all, and which can represent any possible joint probability 
distribution over the given variables. The set DF will contain all possible distribu- 
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Figure 8.25 


TEREPRE 


We can view a graphical model (in this case a directed graph) as a filter in which a prob- 
ability distribution p(x) is allowed through the filter if, and only if, it satisfies the directed 
factorization property (8.5). The set of all possible probability distributions p(x) that pass 
through the filter is denoted DF. We can alternatively use the graph to filter distributions 
according to whether they respect all of the conditional independencies implied by the 
d-separation properties of the graph. The d-separation theorem says that it is the same 
set of distributions DF that will be allowed through this second kind of filter. 


tions p(x). At the other extreme, we have the fully disconnected graph, i.e., one 
having no links at all. This corresponds to joint distributions which factorize into the 
product of the marginal distributions over the variables comprising the nodes of the 
graph. 

Note that for any given graph, the set of distributions DF will include any dis- 
tributions that have additional independence properties beyond those described by 
the graph. For instance, a fully factorized distribution will always be passed through 
the filter implied by any graph over the corresponding set of variables. 

We end our discussion of conditional independence properties by exploring the 
concept of a Markov blanket or Markov boundary. Consider a joint distribution 
p(xX1,..-,Xp) represented by a directed graph having D nodes, and consider the 
conditional distribution of a particular node with variables x; conditioned on all of 
the remaining variables x;z;. Using the factorization property (8.5), we can express 
this conditional distribution in the form 


p(X, es ,Xp) 


IES ...,Xp) dx; 
] [ e«tpa,) 
i 


MICARD dx; 
k 


in which the integral is replaced by a summation in the case of discrete variables. We 
now observe that any factor p(x;,|pa,,) that does not have any functional dependence 
on x; can be taken outside the integral over x;, and will therefore cancel between 
numerator and denominator. The only factors that remain will be the conditional 
distribution p(x;|pa,;) for node x; itself, together with the conditional distributions 
for any nodes X+ such that node x; is in the conditioning set of p(x;,|pa;,), in other 
words for which x; is a parent of x;. The conditional p(x;|pa,) will depend on the 
parents of node x;, whereas the conditionals p(x;|pa,,) will depend on the children 


p(x, | 523} ) 
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Figure 8.26 The Markov blanket of a node x; comprises the set 


Section 8.2 


8.3. 


of parents, children and co-parents of the node. It 
has the property that the conditional distribution of 
x;, conditioned on all the remaining variables in the 
graph, is dependent only on the variables in the 


Markov blanket. Ti 


of x; as well as on the co-parents, in other words variables corresponding to parents 
of node xx other than node x;. The set of nodes comprising the parents, the children 
and the co-parents is called the Markov blanket and is illustrated in Figure 8.26. We 
can think of the Markov blanket of a node x; as being the minimal set of nodes that 
isolates x; from the rest of the graph. Note that it is not sufficient to include only the 
parents and children of node x; because the phenomenon of explaining away means 
that observations of the child nodes will not block paths to the co-parents. We must 
therefore observe the co-parent nodes also. 


Markov Random Fields 


We have seen that directed graphical models specify a factorization of the joint dis- 
tribution over a set of variables into a product of local conditional distributions. They 
also define a set of conditional independence properties that must be satisfied by any 
distribution that factorizes according to the graph. We turn now to the second ma- 
jor class of graphical models that are described by undirected graphs and that again 
specify both a factorization and a set of conditional independence relations. 

A Markov random field, also known as a Markov network or an undirected 
graphical model (Kindermann and Snell, 1980), has a set of nodes each of which 
corresponds to a variable or group of variables, as well as a set of links each of 
which connects a pair of nodes. The links are undirected, that is they do not carry 
arrows. In the case of undirected graphs, it is convenient to begin with a discussion 
of conditional independence properties. 


8.3.1 Conditional independence properties 


In the case of directed graphs, we saw that it was possible to test whether a par- 
ticular conditional independence property holds by applying a graphical test called 
d-separation. This involved testing whether or not the paths connecting two sets of 
nodes were ‘blocked’. The definition of blocked, however, was somewhat subtle 
due to the presence of paths having head-to-head nodes. We might ask whether it 
is possible to define an alternative graphical semantics for probability distributions 
such that conditional independence is determined by simple graph separation. This 
is indeed the case and corresponds to undirected graphical models. By removing the 
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Figure 8.27 An example of an undirected graph in 
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which every path from any node in set 
A to any node in set B passes through 
at least one node in set C. Conse- 
quently the conditional independence 
property A I! B | C holds for any ; 
probability distribution described by this 
graph. 


oerr--un 
==. 


directionality from the links of the graph, the asymmetry between parent and child 
nodes is removed, and so the subtleties associated with head-to-head nodes no longer 
arise. 

Suppose that in an undirected graph we identify three sets of nodes, denoted A, 
B, and C, and that we consider the conditional independence property 


AIL BIC. (8.37) 


To test whether this property is satisfied by a probability distribution defined by a 
graph we consider all possible paths that connect nodes in set A to nodes in set 
B. If all such paths pass through one or more nodes in set C', then all such paths are 
‘blocked’ and so the conditional independence property holds. However, if there is at 
least one such path that is not blocked, then the property does not necessarily hold, or 
more precisely there will exist at least some distributions corresponding to the graph 
that do not satisfy this conditional independence relation. This is illustrated with an 
example in Figure 8.27. Note that this is exactly the same as the d-separation crite- 
rion except that there is no ‘explaining away’ phenomenon. Testing for conditional 
independence in undirected graphs is therefore simpler than in directed graphs. 

An alternative way to view the conditional independence test is to imagine re- 
moving all nodes in set C from the graph together with any links that connect to 
those nodes. We then ask if there exists a path that connects any node in A to any 
node in B. If there are no such paths, then the conditional independence property 
must hold. 

The Markov blanket for an undirected graph takes a particularly simple form, 
because a node will be conditionally independent of all other nodes conditioned only 
on the neighbouring nodes, as illustrated in Figure 8.28. 


8.3.2 Factorization properties 


We now seek a factorization rule for undirected graphs that will correspond to 
the above conditional independence test. Again, this will involve expressing the joint 
distribution p(x) as a product of functions defined over sets of variables that are local 
to the graph. We therefore need to decide what is the appropriate notion of locality 
in this case. 


Figure 8.28 


Figure 8.29 
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For an undirected graph, the Markov blanket of a node 
x; consists of the set of neighbouring nodes. It has the 
property that the conditional distribution of «;, conditioned 
on all the remaining variables in the graph, is dependent 
only on the variables in the Markov blanket. 


If we consider two nodes x; and z; that are not connected by a link, then these 
variables must be conditionally independent given all other nodes in the graph. This 
follows from the fact that there is no direct path between the two nodes, and all other 
paths pass through nodes that are observed, and hence those paths are blocked. This 
conditional independence property can be expressed as 


pti, £5|X\ 44,53) = P( Tal X\ (4,53) P( 25 1X iy) (8.38) 


where x\ ¢;,;} denotes the set x of all variables with x; and x; removed. The factor- 
ization of the joint distribution must therefore be such that x; and x; do not appear 
in the same factor in order for the conditional independence property to hold for all 
possible distributions belonging to the graph. 

This leads us to consider a graphical concept called a clique, which is defined 
as a subset of the nodes in a graph such that there exists a link between all pairs of 
nodes in the subset. In other words, the set of nodes in a clique is fully connected. 
Furthermore, a maximal clique is a clique such that it is not possible to include any 
other nodes from the graph in the set without it ceasing to be a clique. These concepts 
are illustrated by the undirected graph over four variables shown in Figure 8.29. This 
graph has five cliques of two nodes given by {21,22}, {%2, £3}, {£3, va}, {@4, £2}, 
and {x1, 73}, as well as two maximal cliques given by {x1, £2, 73} and {x2, £3, £4}. 
The set {x1, £2, x3, x4} is not a clique because of the missing link from x to x4. 

We can therefore define the factors in the decomposition of the joint distribution 
to be functions of the variables in the cliques. In fact, we can consider functions 
of the maximal cliques, without loss of generality, because other cliques must be 
subsets of maximal cliques. Thus, if {2, £2, 23} is a maximal clique and we define 
an arbitrary function over this clique, then including another factor defined over a 
subset of these variables would be redundant. 

Let us denote a clique by C and the set of variables in that clique by xc. Then 


A four-node undirected graph showing a clique (outlined in 
green) and a maximal clique (outlined in blue). 
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the joint distribution is written as a product of potential functions yc(xc) over the 
maximal cliques of the graph 


1 
p(x) = = |] ve(xc). (8.39) 
c 
Here the quantity Z, sometimes called the partition function, is a normalization con- 


stant and is given by 
Z=% | [vc*co) (8.40) 
x C 


which ensures that the distribution p(x) given by (8.39) is correctly normalized. 
By considering only potential functions which satisfy Yc(xc) > 0 we ensure that 
p(x) > 0. In (8.40) we have assumed that x comprises discrete variables, but the 
framework is equally applicable to continuous variables, or a combination of the two, 
in which the summation is replaced by the appropriate combination of summation 
and integration. 

Note that we do not restrict the choice of potential functions to those that have a 
specific probabilistic interpretation as marginal or conditional distributions. This is 
in contrast to directed graphs in which each factor represents the conditional distribu- 
tion of the corresponding variable, conditioned on the state of its parents. However, 
in special cases, for instance where the undirected graph is constructed by starting 
with a directed graph, the potential functions may indeed have such an interpretation, 
as we shall see shortly. 

One consequence of the generality of the potential functions Wc(xc) is that 
their product will in general not be correctly normalized. We therefore have to in- 
troduce an explicit normalization factor given by (8.40). Recall that for directed 
graphs, the joint distribution was automatically normalized as a consequence of the 
normalization of each of the conditional distributions in the factorization. 

The presence of this normalization constant is one of the major limitations of 
undirected graphs. If we have a model with M discrete nodes each having K states, 
then the evaluation of the normalization term involves summing over K™ states and 
so (in the worst case) is exponential in the size of the model. The partition function 
is needed for parameter learning because it will be a function of any parameters that 
govern the potential functions Yc (xc). However, for evaluation of local conditional 
distributions, the partition function is not needed because a conditional is the ratio 
of two marginals, and the partition function cancels between numerator and denom- 
inator when evaluating this ratio. Similarly, for evaluating local marginal probabil- 
ities we can work with the unnormalized joint distribution and then normalize the 
marginals explicitly at the end. Provided the marginals only involves a small number 
of variables, the evaluation of their normalization coefficient will be feasible. 

So far, we have discussed the notion of conditional independence based on sim- 
ple graph separation and we have proposed a factorization of the joint distribution 
that is intended to correspond to this conditional independence structure. However, 
we have not made any formal connection between conditional independence and 
factorization for undirected graphs. To do so we need to restrict attention to poten- 
tial functions Wc (xc) that are strictly positive (i.e., never zero or negative for any 
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choice of xc). Given this restriction, we can make a precise relationship between 
factorization and conditional independence. 

To do this we again return to the concept of a graphical model as a filter, corre- 
sponding to Figure 8.25. Consider the set of all possible distributions defined over 
a fixed set of variables corresponding to the nodes of a particular undirected graph. 
We can define UZ to be the set of such distributions that are consistent with the set 
of conditional independence statements that can be read from the graph using graph 
separation. Similarly, we can define UF to be the set of such distributions that can 
be expressed as a factorization of the form (8.39) with respect to the maximal cliques 
of the graph. The Hammersley-Clifford theorem (Clifford, 1990) states that the sets 
UT and UF are identical. 

Because we are restricted to potential functions which are strictly positive it is 
convenient to express them as exponentials, so that 


Wc(xc) = exp {-—E(xc)} (8.41) 


where E(xc) is called an energy function, and the exponential representation is 
called the Boltzmann distribution. The joint distribution is defined as the product of 
potentials, and so the total energy is obtained by adding the energies of each of the 
maximal cliques. 

In contrast to the factors in the joint distribution for a directed graph, the po- 
tentials in an undirected graph do not have a specific probabilistic interpretation. 
Although this gives greater flexibility in choosing the potential functions, because 
there is no normalization constraint, it does raise the question of how to motivate a 
choice of potential function for a particular application. This can be done by view- 
ing the potential function as expressing which configurations of the local variables 
are preferred to others. Global configurations that have a relatively high probability 
are those that find a good balance in satisfying the (possibly conflicting) influences 
of the clique potentials. We turn now to a specific example to illustrate the use of 
undirected graphs. 


8.3.3 Illustration: Image de-noising 


We can illustrate the application of undirected graphs using an example of noise 
removal from a binary image (Besag, 1974; Geman and Geman, 1984; Besag, 1986). 
Although a very simple example, this is typical of more sophisticated applications. 
Let the observed noisy image be described by an array of binary pixel values y; € 
{—1, +1}, where the index i = 1,..., D runs over all pixels. We shall suppose 
that the image is obtained by taking an unknown noise-free image, described by 
binary pixel values x; € {—1, +1} and randomly flipping the sign of pixels with 
some small probability. An example binary image, together with a noise corrupted 
image obtained by flipping the sign of the pixels with probability 10%, is shown in 
Figure 8.30. Given the noisy image, our goal is to recover the original noise-free 
image. 

Because the noise level is small, we know that there will be a strong correlation 
between x; and y;. We also know that neighbouring pixels x; and x; in an image 
are strongly correlated. This prior knowledge can be captured using the Markov 
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Figure 8.30 


- oa 


Illustration of image de-noising using a Markov random field. The top row shows the original 


binary image on the left and the corrupted image after randomly changing 10% of the pixels on the right. The 
bottom row shows the restored images obtained using iterated conditional models (ICM) on the left and using 
the graph-cut algorithm on the right. ICM produces an image where 96% of the pixels agree with the original 
image, whereas the corresponding number for graph-cut is 99%. 


random field model whose undirected graph is shown in Figure 8.31. This graph has 
two types of cliques, each of which contains two variables. The cliques of the form 
{x;, yi} have an associated energy function that expresses the correlation between 
these variables. We choose a very simple energy function for these cliques of the 
form —nx;y; where 7 is a positive constant. This has the desired effect of giving a 
lower energy (thus encouraging a higher probability) when x; and y; have the same 
sign and a higher energy when they have the opposite sign. 

The remaining cliques comprise pairs of variables {x;,x;} where 7 and j are 
indices of neighbouring pixels. Again, we want the energy to be lower when the 
pixels have the same sign than when they have the opposite sign, and so we choose 
an energy given by —(x;2,; where (3 is a positive constant. 

Because a potential function is an arbitrary, nonnegative function over a maximal 
clique, we can multiply it by any nonnegative functions of subsets of the clique, or 


Figure 8.31 An undirected graphical model representing a 
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Markov random field for image de-noising, in 
which x; is a binary variable denoting the state 
of pixel 2 in the unknown noise-free image, and y; 
denotes the corresponding value of pixel i in the 
observed noisy image. 


equivalently we can add the corresponding energies. In this example, this allows us 
to add an extra term ha, for each pixel 7 in the noise-free image. Such a term has 
the effect of biasing the model towards pixel values that have one particular sign in 
preference to the other. 

The complete energy function for the model then takes the form 


E(x,y)=h s- BY tizj- nY} tiv: (8.42) 


which defines a joint distribution over x and y given by 


1 
p(x, y) = Z exp{—E(x,y)}. (8.43) 


We now fix the elements of y to the observed values given by the pixels of the 
noisy image, which implicitly defines a conditional distribution p(x|y) over noise- 
free images. This is an example of the sing model, which has been widely studied in 
Statistical physics. For the purposes of image restoration, we wish to find an image x 
having a high probability (ideally the maximum probability). To do this we shall use 
a simple iterative technique called iterated conditional modes, or ICM (Kittler and 
Foglein, 1984), which is simply an application of coordinate-wise gradient ascent. 
The idea is first to initialize the variables {x;}, which we do by simply setting x; = 
yi for all i. Then we take one node x; at a time and we evaluate the total energy 
for the two possible states x; = +1 and x; = —1, keeping all other node variables 
fixed, and set x; to whichever state has the lower energy. This will either leave 
the probability unchanged, if xj is unchanged, or will increase it. Because only 
one variable is changed, this is a simple local computation that can be performed 
efficiently. We then repeat the update for another site, and so on, until some suitable 
stopping criterion is satisfied. The nodes may be updated in a systematic way, for 
instance by repeatedly raster scanning through the image, or by choosing nodes at 
random. 

If we have a sequence of updates in which every site is visited at least once, 
and in which no changes to the variables are made, then by definition the algorithm 
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Figure 8.32 (a) Example of a directed Tı T2 TN-—1 TN 


graph. (b) The equivalent undirected (a) O—O— pass —( )}—) 
graph. 
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will have converged to a local maximum of the probability. This need not, however, 
correspond to the global maximum. 

For the purposes of this simple illustration, we have fixed the parameters to be 
GB = 1.0, n = 2.1 and h = 0. Note that leaving h = 0 simply means that the prior 
probabilities of the two states of x; are equal. Starting with the observed noisy image 
as the initial configuration, we run ICM until convergence, leading to the de-noised 
image shown in the lower left panel of Figure 8.30. Note that if we set 8 = 0, 
which effectively removes the links between neighbouring pixels, then the global 
most probable solution is given by x; = y; for all ¿, corresponding to the observed 
noisy image. 

Later we shall discuss a more effective algorithm for finding high probability so- 
lutions called the max-product algorithm, which typically leads to better solutions, 
although this is still not guaranteed to find the global maximum of the posterior dis- 
tribution. However, for certain classes of model, including the one given by (8.42), 
there exist efficient algorithms based on graph cuts that are guaranteed to find the 
global maximum (Greig et al., 1989; Boykov et al., 2001; Kolmogorov and Zabih, 
2004). The lower right panel of Figure 8.30 shows the result of applying a graph-cut 
algorithm to the de-noising problem. 


8.3.4 Relation to directed graphs 


We have introduced two graphical frameworks for representing probability dis- 
tributions, corresponding to directed and undirected graphs, and it is instructive to 
discuss the relation between these. Consider first the problem of taking a model that 
is specified using a directed graph and trying to convert it to an undirected graph. In 
some cases this is straightforward, as in the simple example in Figure 8.32. Here the 
joint distribution for the directed graph is given as a product of conditionals in the 
form 

P(x) = p(21)p(@2|@1)p(x3| x2) «++ p(en|an-1). (8.44) 


Now let us convert this to an undirected graph representation, as shown in Fig- 
ure 8.32. In the undirected graph, the maximal cliques are simply the pairs of neigh- 
bouring nodes, and so from (8.39) we wish to write the joint distribution in the form 


1 
p(x) = z222, T2)Y2,3 (£2, £3) nee Wn-1,N(@N-1, IN). (8.45) 
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Figure 8.33 Example of a simple Tı T3 Tı T3 
directed graph (a) and the corre- 
sponding moral graph (b). T2 


T4 T4 


(a) (b) 


This is easily done by identifying 


Yı 2(£1, £2) = p(zı)p(zə|x1) 
P2 3(£2,£3) = p(x3|x2) 
Wn-1,nN(2nN-1,0N) = plen|zn-1) 


where we have absorbed the marginal p(x) for the first node into the first potential 
function. Note that in this case, the partition function Z = 1. 

Let us consider how to generalize this construction, so that we can convert any 
distribution specified by a factorization over a directed graph into one specified by a 
factorization over an undirected graph. This can be achieved if the clique potentials 
of the undirected graph are given by the conditional distributions of the directed 
graph. In order for this to be valid, we must ensure that the set of variables that 
appears in each of the conditional distributions is a member of at least one clique of 
the undirected graph. For nodes on the directed graph having just one parent, this is 
achieved simply by replacing the directed link with an undirected link. However, for 
nodes in the directed graph having more than one parent, this is not sufficient. These 
are nodes that have ‘head-to-head’ paths encountered in our discussion of conditional 
independence. Consider a simple directed graph over 4 nodes shown in Figure 8.33. 
The joint distribution for the directed graph takes the form 


p(x) = p(xı)p(x2)p(x3)p(xa|x£1, £2, £3). (8.46) 


We see that the factor p(x4|£1, £2, £3) involves the four variables z1, £2, £3, and 
x4, and so these must all belong to a single clique if this conditional distribution is 
to be absorbed into a clique potential. To ensure this, we add extra links between 
all pairs of parents of the node x4. Anachronistically, this process of ‘marrying 
the parents’ has become known as moralization, and the resulting undirected graph, 
after dropping the arrows, is called the moral graph. It is important to observe that 
the moral graph in this example is fully connected and so exhibits no conditional 
independence properties, in contrast to the original directed graph. 

Thus in general to convert a directed graph into an undirected graph, we first add 
additional undirected links between all pairs of parents for each node in the graph and 
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Figure 8.34 


then drop the arrows on the original links to give the moral graph. Then we initialize 
all of the clique potentials of the moral graph to 1. We then take each conditional 
distribution factor in the original directed graph and multiply it into one of the clique 
potentials. There will always exist at least one maximal clique that contains all of 
the variables in the factor as a result of the moralization step. Note that in all cases 
the partition function is given by Z = 1. 

The process of converting a directed graph into an undirected graph plays an 
important role in exact inference techniques such as the junction tree algorithm. 
Converting from an undirected to a directed representation is much less common 
and in general presents problems due to the normalization constraints. 

We saw that in going from a directed to an undirected representation we had to 
discard some conditional independence properties from the graph. Of course, we 
could always trivially convert any distribution over a directed graph into one over an 
undirected graph by simply using a fully connected undirected graph. This would, 
however, discard all conditional independence properties and so would be vacuous. 
The process of moralization adds the fewest extra links and so retains the maximum 
number of independence properties. 

We have seen that the procedure for determining the conditional independence 
properties is different between directed and undirected graphs. It turns out that the 
two types of graph can express different conditional independence properties, and 
it is worth exploring this issue in more detail. To do so, we return to the view of 
a specific (directed or undirected) graph as a filter, so that the set of all possible 
distributions over the given variables could be reduced to a subset that respects the 
conditional independencies implied by the graph. A graph is said to be a D map 
(for ‘dependency map’) of a distribution if every conditional independence statement 
satisfied by the distribution is reflected in the graph. Thus a completely disconnected 
graph (no links) will be a trivial D map for any distribution. 

Alternatively, we can consider a specific distribution and ask which graphs have 
the appropriate conditional independence properties. If every conditional indepen- 
dence statement implied by a graph is satisfied by a specific distribution, then the 
graph is said to be an J map (for ‘independence map’) of that distribution. Clearly a 
fully connected graph will be a trivial I map for any distribution. 

If it is the case that every conditional independence property of the distribution 
is reflected in the graph, and vice versa, then the graph is said to be a perfect map for 


Venn diagram illustrating the set of all distributions 
P over a given set of variables, together with the 
set of distributions D that can be represented as a 
perfect map using a directed graph, and the set U 
that can be represented as a perfect map using an 
undirected graph. 


Figure 8.35 


Figure 8.36 


8.4. 


8.4. Inference in Graphical Models 393 


A directed graph whose conditional independence A B 
properties cannot be expressed using an undirected 
graph over the same three variables. 


that distribution. A perfect map is therefore both an I map and a D map. 

Consider the set of distributions such that for each distribution there exists a 
directed graph that is a perfect map. This set is distinct from the set of distributions 
such that for each distribution there exists an undirected graph that is a perfect map. 
In addition there are distributions for which neither directed nor undirected graphs 
offer a perfect map. This is illustrated as a Venn diagram in Figure 8.34. 

Figure 8.35 shows an example of a directed graph that is a perfect map for 
a distribution satisfying the conditional independence properties A 1L B | Ø and 
A B |C. There is no corresponding undirected graph over the same three vari- 
ables that is a perfect map. 

Conversely, consider the undirected graph over four variables shown in Fig- 
ure 8.36. This graph exhibits the properties AM B | 0, C IL D| AU B and 
A 1L B | CUD. There is no directed graph over four variables that implies the same 
set of conditional independence properties. 

The graphical framework can be extended in a consistent way to graphs that 
include both directed and undirected links. These are called chain graphs (Lauritzen 
and Wermuth, 1989; Frydenberg, 1990), and contain the directed and undirected 
graphs considered so far as special cases. Although such graphs can represent a 
broader class of distributions than either directed or undirected alone, there remain 
distributions for which even a chain graph cannot provide a perfect map. Chain 
graphs are not discussed further in this book. 


An undirected graph whose conditional independence C 
properties cannot be expressed in terms of a directed 
graph over the same variables. 


Inference in Graphical Models 


We turn now to the problem of inference in graphical models, in which some of 
the nodes in a graph are clamped to observed values, and we wish to compute the 
posterior distributions of one or more subsets of other nodes. As we shall see, we 
can exploit the graphical structure both to find efficient algorithms for inference, and 
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Figure 8.37 A graphical representation of Bayes’ theorem. 2 x x 


See the text for details. 


to make the structure of those algorithms transparent. Specifically, we shall see that 
many algorithms can be expressed in terms of the propagation of local messages 
around the graph. In this section, we shall focus primarily on techniques for exact 
inference, and in Chapter 10 we shall consider a number of approximate inference 
algorithms. 

To start with, let us consider the graphical interpretation of Bayes’ theorem. 
Suppose we decompose the joint distribution p(x, y) over two variables x and y into 
a product of factors in the form p(x, y) = p(x)p(y|x). This can be represented by 
the directed graph shown in Figure 8.37(a). Now suppose we observe the value of 
y, as indicated by the shaded node in Figure 8.37(b). We can view the marginal 
distribution p(x) as a prior over the latent variable x, and our goal is to infer the 
corresponding posterior distribution over x. Using the sum and product rules of 
probability we can evaluate 


ply) = X` vlyla’)p(2') (8.47) 
which can then be used in Bayes’ theorem to calculate 
x) p(x 
pea. (8.48) 
p(y) 


Thus the joint distribution is now expressed in terms of p(y) and p(x|y). From a 
graphical perspective, the joint distribution p(x, y) is now represented by the graph 
shown in Figure 8.37(c), in which the direction of the arrow is reversed. This is the 
simplest example of an inference problem for a graphical model. 


8.4.1 Inference on a chain 


Now consider a more complex problem involving the chain of nodes of the form 
shown in Figure 8.32. This example will lay the foundation for a discussion of exact 
inference in more general graphs later in this section. 

Specifically, we shall consider the undirected graph in Figure 8.32(b). We have 
already seen that the directed chain can be transformed into an equivalent undirected 
chain. Because the directed graph does not have any nodes with more than one 
parent, this does not require the addition of any extra links, and the directed and 
undirected versions of this graph express exactly the same set of conditional inde- 
pendence statements. 
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The joint distribution for this graph takes the form 


1 
p(x) = gv 12 (81, £2 )¥2,3(L2, £3) +++ Wn-1,n(€@N-1, EN). (8.49) 


We shall consider the specific case in which the N nodes represent discrete vari- 
ables each having K states, in which case each potential function Yn—1,n(Zn—1, Ln) 
comprises an K x K table, and so the joint distribution has (N — 1) K? parameters. 

Let us consider the inference problem of finding the marginal distribution p(£n ) 
for a specific node x,, that is part way along the chain. Note that, for the moment, 
there are no observed nodes. By definition, the required marginal is obtained by 
summing the joint distribution over all variables except xn, so that 


Plan) =X 5 Sos Swe). (8.50) 


TLn—1 Tn+1 


In a naive implementation, we would first evaluate the joint distribution and 
then perform the summations explicitly. The joint distribution can be represented as 
a set of numbers, one for each possible value for x. Because there are N variables 
each with K states, there are KY values for x and so evaluation and storage of the 
joint distribution, as well as marginalization to obtain p(z,,), all involve storage and 
computation that scale exponentially with the length N of the chain. 

We can, however, obtain a much more efficient algorithm by exploiting the con- 
ditional independence properties of the graphical model. If we substitute the factor- 
ized expression (8.49) for the joint distribution into (8.50), then we can rearrange the 
order of the summations and the multiplications to allow the required marginal to be 
evaluated much more efficiently. Consider for instance the summation over x y. The 
potential Yn—1,N(£N—1, zN) is the only one that depends on xy, and so we can 
perform the summation 


X by-1,.y(en-1, tN) (8.51) 


first to give a function of £y—1ı. We can then use this to perform the summation 
over £y—1, Which will involve only this new function together with the potential 
Wn—2,N—1(@N—2,%N-—1), because this is the only other place that x j_1 appears. 
Similarly, the summation over x; involves only the potential ~1,2(@1, 22) and so 
can be performed separately to give a function of x2, and so on. Because each 
summation effectively removes a variable from the distribution, this can be viewed 
as the removal of a node from the graph. 

If we group the potentials and summations together in this way, we can express 
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the desired marginal in the form 


pan) = 5 
5 Yn—1,n(Tn—1; Tn) a £ Wa 3(£2, £3) b balena) g 


HalTn) 


2 Wn pad tin Tn41) oes £ Wn-1,N(LN-1, ex) aca as (8.52) 


Ln+1 


Lg (Xn) 


The reader is encouraged to study this re-ordering carefully as the underlying idea 
forms the basis for the later discussion of the general sum-product algorithm. Here 
the key concept that we are exploiting is that multiplication is distributive over addi- 
tion, so that 

ab+ac=a(b+c) (8.53) 


in which the left-hand side involves three arithmetic operations whereas the right- 
hand side reduces this to two operations. 

Let us work out the computational cost of evaluating the required marginal using 
this re-ordered expression. We have to perform N — 1 summations each of which is 
over K states and each of which involves a function of two variables. For instance, 
the summation over x, involves only the function Yı 2(£1, £2), which is a table of 
K x K numbers. We have to sum this table over x, for each value of x2 and so this 
has O(K?°) cost. The resulting vector of K numbers is multiplied by the matrix of 
numbers w2,3(22, 73) and so is again O(K?). Because there are N — 1 summations 
and multiplications of this kind, the total cost of evaluating the marginal p(z,,) is 
O(N K?). This is linear in the length of the chain, in contrast to the exponential cost 
of a naive approach. We have therefore been able to exploit the many conditional 
independence properties of this simple graph in order to obtain an efficient calcula- 
tion. If the graph had been fully connected, there would have been no conditional 
independence properties, and we would have been forced to work directly with the 
full joint distribution. 

We now give a powerful interpretation of this calculation in terms of the passing 
of local messages around on the graph. From (8.52) we see that the expression for the 
marginal p(x,,) decomposes into the product of two factors times the normalization 
constant 


1 
p(X) = ghaltn)Ha(tn): (8.54) 


We shall interpret Ha(£n) as a message passed forwards along the chain from node 
Tn—ı to node £n. Similarly, j1g(x,,) can be viewed as a message passed backwards 
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Figure 8.38 The marginal distribution He(fn-1) Haltn) a(@n)  HplEn+1) 

p(%n) for a node xn along the chain is ob- — — + — 

tained by multiplying the two messages C+ selen —C)}—C)}—C_ + eer —C) 
Hal(£n) and ug(£n), and then normaliz- 

ing. These messages can themselves 71 Tn—-1 Tn Tn+1 UN 


be evaluated recursively by passing mes- 
sages from both ends of the chain to- 
wards node zn. 


along the chain to node x, from node x,,,,. Note that each of the messages com- 
prises a set of K values, one for each choice of £n, and so the product of two mes- 
sages should be interpreted as the point-wise multiplication of the elements of the 
two messages to give another set of values. 

The message //q(Z,,) can be evaluated recursively because 


Ma(fn) = S hent I 


= S ie ala) (8.55) 
We therefore first evaluate 
Halz2) = X Y1,2(21, 22) (8.56) 


and then apply (8.55) repeatedly until we reach the desired node. Note carefully the 
structure of the message passing equation. The outgoing message Ha(£n) in (8.55) 
is obtained by multiplying the incoming message [1a (a1) by the local potential 
involving the node variable and the outgoing variable and then summing over the 
node variable. 

Similarly, the message j1g(x,,) can be evaluated recursively by starting with 
node xy and using 


Up(Ln) = ` VUntrnl Intis Tn) E iiis 
Tn+1 Tn4+2 
= Ls Wd scl Pads Tq) fig Gnd). (8.57) 


This recursive message passing is illustrated in Figure 8.38. The normalization con- 
stant Z is easily evaluated by summing the right-hand side of (8.54) over all states 
of zn, an operation that requires only O(K) computation. 

Graphs of the form shown in Figure 8.38 are called Markov chains, and the 
corresponding message passing equations represent an example of the Chapman- 
Kolmogorov equations for Markov processes (Papoulis, 1984). 
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Now suppose we wish to evaluate the marginals p(x,,) for every node n € 
{1,..., N} in the chain. Simply applying the above procedure separately for each 
node will have computational cost that is O(N? M?). However, such an approach 
would be very wasteful of computation. For instance, to find p(x) we need to prop- 
agate a message j1g(-) from node xy back to node x2. Similarly, to evaluate p(x2) 
we need to propagate a messages j1g(-) from node xy back to node x3. This will 
involve much duplicated computation because most of the messages will be identical 
in the two cases. 

Suppose instead we first launch a message ug(xy-—1) starting from node xy 
and propagate corresponding messages all the way back to node x, and suppose we 
similarly launch a message 4a(x2) starting from node x, and propagate the corre- 
sponding messages all the way forward to node xy. Provided we store all of the 
intermediate messages along the way, then any node can evaluate its marginal sim- 
ply by applying (8.54). The computational cost is only twice that for finding the 
marginal of a single node, rather than N times as much. Observe that a message 
has passed once in each direction across each link in the graph. Note also that the 
normalization constant Z need be evaluated only once, using any convenient node. 

If some of the nodes in the graph are observed, then the corresponding variables 
are simply clamped to their observed values and there is no summation. To see 
this, note that the effect of clamping a variable x, to an observed value 7, can 
be expressed by multiplying the joint distribution by (one or more copies of) an 
additional function I(x,,,Z,,), which takes the value 1 when £n = @,, and the value 
0 otherwise. One such function can then be absorbed into each of the potentials that 
contain £n. Summations over £n then contain only one term in which £n = Tn. 

Now suppose we wish to calculate the joint distribution p(£n—1, £n) for two 
neighbouring nodes on the chain. This is similar to the evaluation of the marginal 
for a single node, except that there are now two variables that are not summed out. 
A few moments thought will show that the required joint distribution can be written 
in the form 


1 
Dina is Tn) = gheltn—-1)bn—1,0(n-1, en ia fn) (8.58) 


Thus we can obtain the joint distributions over all of the sets of variables in each 
of the potentials directly once we have completed the message passing required to 
obtain the marginals. 

This is a useful result because in practice we may wish to use parametric forms 
for the clique potentials, or equivalently for the conditional distributions if we started 
from a directed graph. In order to learn the parameters of these potentials in situa- 
tions where not all of the variables are observed, we can employ the EM algorithm, 
and it turns out that the local joint distributions of the cliques, conditioned on any 
observed data, is precisely what is needed in the E step. We shall consider some 
examples of this in detail in Chapter 13. 


8.4.2 Trees 


We have seen that exact inference on a graph comprising a chain of nodes can be 
performed efficiently in time that is linear in the number of nodes, using an algorithm 


8.4. Inference in Graphical Models 399 


Figure 8.39 Examples of tree- 
structured graphs, showing (a) an 
undirected tree, (b) a directed tree, 
and (c) a directed polytree. 
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(a) (b) (c) 


that can be interpreted in terms of messages passed along the chain. More generally, 
inference can be performed efficiently using local message passing on a broader 
class of graphs called trees. In particular, we shall shortly generalize the message 
passing formalism derived above for chains to give the sum-product algorithm, which 
provides an efficient framework for exact inference in tree-structured graphs. 

In the case of an undirected graph, a tree is defined as a graph in which there 
is one, and only one, path between any pair of nodes. Such graphs therefore do not 
have loops. In the case of directed graphs, a tree is defined such that there is a single 
node, called the root, which has no parents, and all other nodes have one parent. If 
we convert a directed tree into an undirected graph, we see that the moralization step 
will not add any links as all nodes have at most one parent, and as a consequence the 
corresponding moralized graph will be an undirected tree. Examples of undirected 
and directed trees are shown in Figure 8.39(a) and 8.39(b). Note that a distribution 
represented as a directed tree can easily be converted into one represented by an 
undirected tree, and vice versa. 

If there are nodes in a directed graph that have more than one parent, but there is 
still only one path (ignoring the direction of the arrows) between any two nodes, then 
the graph is a called a polytree, as illustrated in Figure 8.39(c). Such a graph will 
have more than one node with the property of having no parents, and furthermore, 
the corresponding moralized undirected graph will have loops. 


8.4.3 Factor graphs 


The sum-product algorithm that we derive in the next section is applicable to 
undirected and directed trees and to polytrees. It can be cast in a particularly simple 
and general form if we first introduce a new graphical construction called a factor 
graph (Frey, 1998; Kschischnang et al., 2001). 

Both directed and undirected graphs allow a global function of several vari- 
ables to be expressed as a product of factors over subsets of those variables. Factor 
graphs make this decomposition explicit by introducing additional nodes for the fac- 
tors themselves in addition to the nodes representing the variables. They also allow 
us to be more explicit about the details of the factorization, as we shall see. 

Let us write the joint distribution over a set of variables in the form of a product 


of factors 
p(x) = II f(x) (8.59) 


where x, denotes a subset of the variables. For convenience, we shall denote the 
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Figure 8.40 Example of a factor graph, which corresponds Tı T2 T3 


Figure 8.41 


(a) 


T3 


to the factorization (8.60). 


fa fo fe fa 


individual variables by x;, however, as in earlier discussions, these can comprise 
groups of variables (such as vectors or matrices). Each factor fs is a function of a 
corresponding set of variables xs. 

Directed graphs, whose factorization is defined by (8.5), represent special cases 
of (8.59) in which the factors f,(x,) are local conditional distributions. Similarly, 
undirected graphs, given by (8.39), are a special case in which the factors are po- 
tential functions over the maximal cliques (the normalizing coefficient 1/Z can be 
viewed as a factor defined over the empty set of variables). 

In a factor graph, there is a node (depicted as usual by a circle) for every variable 
in the distribution, as was the case for directed and undirected graphs. There are also 
additional nodes (depicted by small squares) for each factor f,(x,) in the joint dis- 
tribution. Finally, there are undirected links connecting each factor node to all of the 
variables nodes on which that factor depends. Consider, for example, a distribution 
that is expressed in terms of the factorization 


p(x) = fa(t1, £2) fo(£1, £2) fe(£2, £3) falas). (8.60) 


This can be expressed by the factor graph shown in Figure 8.40. Note that there are 
two factors fa(£1, £2) and f,(x1, x2) that are defined over the same set of variables. 
In an undirected graph, the product of two such factors would simply be lumped 
together into the same clique potential. Similarly, fe(x2, x3) and fy(x3) could be 
combined into a single potential over x2 and x3. The factor graph, however, keeps 
such factors explicit and so is able to convey more detailed information about the 
underlying factorization. 


T2 Ly T2 Tı T2 


fo 
T3 T3 


(b) (c) 


(a) An undirected graph with a single clique potential Y(x1, x2, x3). (b) A factor graph with factor 
f(z1,£2,£3) = Y(z1, £2, £3) representing the same distribution as the undirected graph. (c) A different factor 
graph representing the same distribution, whose factors satisfy fa(£1, £2, £3) fol(£1, £2) = Y(z£1, £2, £3). 


T3 


(a) 
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T2 Tı T2 Tı T2 


T3 T3 


(b) (c) 


Figure 8.42 (a) A directed graph with the factorization p(xı)p(x2)p(x3|xı, x2). (b) A factor graph representing 
the same distribution as the directed graph, whose factor satisfies f(x1, £2, x3) = p(x1)p(x2)p(x3|x1, £2). (C) 
A different factor graph representing the same distribution with factors fa(v1) = p(x1), fo(v2) = p(x2) and 
fe(1, £2, £3) = p(x3|x1, £2). 


Factor graphs are said to be bipartite because they consist of two distinct kinds 
of nodes, and all links go between nodes of opposite type. In general, factor graphs 
can therefore always be drawn as two rows of nodes (variable nodes at the top and 
factor nodes at the bottom) with links between the rows, as shown in the example in 
Figure 8.40. In some situations, however, other ways of laying out the graph may 
be more intuitive, for example when the factor graph is derived from a directed or 
undirected graph, as we shall see. 

If we are given a distribution that is expressed in terms of an undirected graph, 
then we can readily convert it to a factor graph. To do this, we create variable nodes 
corresponding to the nodes in the original undirected graph, and then create addi- 
tional factor nodes corresponding to the maximal cliques xs. The factors f(x.) are 
then set equal to the clique potentials. Note that there may be several different factor 
graphs that correspond to the same undirected graph. These concepts are illustrated 
in Figure 8.41. 

Similarly, to convert a directed graph to a factor graph, we simply create variable 
nodes in the factor graph corresponding to the nodes of the directed graph, and then 
create factor nodes corresponding to the conditional distributions, and then finally 
add the appropriate links. Again, there can be multiple factor graphs all of which 
correspond to the same directed graph. The conversion of a directed graph to a 
factor graph is illustrated in Figure 8.42. 

We have already noted the importance of tree-structured graphs for performing 
efficient inference. If we take a directed or undirected tree and convert it into a factor 
graph, then the result will again be a tree (in other words, the factor graph will have 
no loops, and there will be one and only one path connecting any two nodes). In 
the case of a directed polytree, conversion to an undirected graph results in loops 
due to the moralization step, whereas conversion to a factor graph again results in a 
tree, as illustrated in Figure 8.43. In fact, local cycles in a directed graph due to 
links connecting parents of a node can be removed on conversion to a factor graph 
by defining the appropriate factor function, as shown in Figure 8.44. 

We have seen that multiple different factor graphs can represent the same di- 
rected or undirected graph. This allows factor graphs to be more specific about the 
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(a) 


(b) (c) 


Figure 8.43 (a) A directed polytree. (b) The result of converting the polytree into an undirected graph showing 
the creation of loops. (c) The result of converting the polytree into a factor graph, which retains the tree structure. 
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Figure 8.44 


precise form of the factorization. Figure 8.45 shows an example of a fully connected 
undirected graph along with two different factor graphs. In (b), the joint distri- 
bution is given by a general form p(x) = f(z£1, £2, £3), whereas in (c), it is given 
by the more specific factorization p(x) = fa(x1, £2) fe (x1, £3) fe(£2, £3). It should 
be emphasized that the factorization in (c) does not correspond to any conditional 
independence properties. 


8.4.4 The sum-product algorithm 


We shall now make use of the factor graph framework to derive a powerful class 
of efficient, exact inference algorithms that are applicable to tree-structured graphs. 
Here we shall focus on the problem of evaluating local marginals over nodes or 
subsets of nodes, which will lead us to the sum-product algorithm. Later we shall 
modify the technique to allow the most probable state to be found, giving rise to the 
max-sum algorithm. 

Also we shall suppose that all of the variables in the model are discrete, and 
so marginalization corresponds to performing sums. The framework, however, is 
equally applicable to linear-Gaussian models in which case marginalization involves 
integration, and we shall consider an example of this in detail when we discuss linear 
dynamical systems. 


(a) A fragment of a di- 1 T2 Tı T2 
rected graph having a lo- 

cal cycle. (b) Conversion 

to a fragment of a factor 

graph having a tree struc- f (#1, £2, £3) 
ture, in which f (x1, £2, £3) = 

p(zı)p(x2|£1)p(z3|x£1, £2). 


T3 T3 
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Tı T2 Tı T2 Tı fo T2 


f(21, £2, £3) 


T3 T3 T3 
(a) (b) (c) 


Figure 8.45 (a) A fully connected undirected graph. (b) and (c) Two factor graphs each of which corresponds 
to the undirected graph in (a). 


There is an algorithm for exact inference on directed graphs without loops known 
as belief propagation (Pearl, 1988; Lauritzen and Spiegelhalter, 1988), and is equiv- 
alent to a special case of the sum-product algorithm. Here we shall consider only the 
sum-product algorithm because it is simpler to derive and to apply, as well as being 
more general. 

We shall assume that the original graph is an undirected tree or a directed tree or 
polytree, so that the corresponding factor graph has a tree structure. We first convert 
the original graph into a factor graph so that we can deal with both directed and 
undirected models using the same framework. Our goal is to exploit the structure of 
the graph to achieve two things: (i) to obtain an efficient, exact inference algorithm 
for finding marginals; (ii) in situations where several marginals are required to allow 
computations to be shared efficiently. 

We begin by considering the problem of finding the marginal p(x) for partic- 
ular variable node x. For the moment, we shall suppose that all of the variables 
are hidden. Later we shall see how to modify the algorithm to incorporate evidence 
corresponding to observed variables. By definition, the marginal is obtained by sum- 
ming the joint distribution over all variables except x so that 


plz) = X` p(x) (8.61) 


x\x 


where x \ z denotes the set of variables in x with variable x omitted. The idea is 
to substitute for p(x) using the factor graph expression (8.59) and then interchange 
summations and products in order to obtain an efficient algorithm. Consider the 
fragment of graph shown in Figure 8.46 in which we see that the tree structure of 
the graph allows us to partition the factors in the joint distribution into groups, with 
one group associated with each of the factor nodes that is a neighbour of the variable 
node x. We see that the joint distribution can be written as a product of the form 


p(x)= |] F(a, X.) (8.62) 
s€ne(x) 


ne(x) denotes the set of factor nodes that are neighbours of x, and X, denotes the 
set of all variables in the subtree connected to the variable node x via the factor node 
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Figure 8.46 A fragment of a factor graph illustrating the 


evaluation of the marginal p(x). 


F(z, Xs) 


fs, and F’,(a, Xs) represents the product of all the factors in the group associated 
with factor fs. 

Substituting (8.62) into (8.61) and interchanging the sums and products, we ob- 
tain 


p(x) =< II Trte xo) 
sEne(x) L Xs 


= |] ae. (8.63) 


s€ne(x) 


Here we have introduced a set of functions ,—.2 (a), defined by 


Lf, 30(t) = X F(a, Xs) (8.64) 
Xs 


which can be viewed as messages from the factor nodes fs to the variable node z. 
We see that the required marginal p(x) is given by the product of all the incoming 
messages arriving at node x. 

In order to evaluate these messages, we again turn to Figure 8.46 and note that 
each factor F(x, Xs) is described by a factor (sub-)graph and so can itself be fac- 
torized. In particular, we can write 


Fall, Xs) = f(x, £1,- ..,£M)G1 (£1, An) -Gm (£m, Xsm) (8.65) 


where, for convenience, we have denoted the variables associated with factor fz, in 

addition to x, by 71,...,@,¢. This factorization is illustrated in Figure 8.47. Note 

that the set of variables {x, x1, ..., xm } is the set of variables on which the factor 

fs depends, and so it can also be denoted xs, using the notation of (8.59). 
Substituting (8.65) into (8.64) we obtain 


fils) = Se Yel ye) II 


zı cM m€ne(f,)\x 


5 ads T | II fee pean) (8.66) 


zı cM m€ne(f,)\x 


rp Gm Xan) 


Xam 


v 
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Figure 8.47 Illustration of the factorization of the subgraph as- 7M 
sociated with factor node fs. 


~~ (£m) 


Tm 


Grin a) 


where ne( fs) denotes the set of variable nodes that are neighbours of the factor node 
fs, and ne( fs) \ x denotes the same set but with node x removed. Here we have 
defined the following messages from variable nodes to factor nodes 


Ham—fs (Em) = 5 Gm(Em, Xam): (8.67) 
X 


sm 


We have therefore introduced two distinct kinds of message, those that go from factor 
nodes to variable nodes denoted 4f—«(x), and those that go from variable nodes to 
factor nodes denoted ju,—. f(x). In each case, we see that messages passed along a 
link are always a function of the variable associated with the variable node that link 
connects to. 

The result (8.66) says that to evaluate the message sent by a factor node to a vari- 
able node along the link connecting them, take the product of the incoming messages 
along all other links coming into the factor node, multiply by the factor associated 
with that node, and then marginalize over all of the variables associated with the 
incoming messages. This is illustrated in Figure 8.47. It is important to note that 
a factor node can send a message to a variable node once it has received incoming 
messages from all other neighbouring variable nodes. 

Finally, we derive an expression for evaluating the messages from variable nodes 
to factor nodes, again by making use of the (sub-)graph factorization. From Fig- 
ure 8.48, we see that term Gm(£m, Xsm) associated with node £m is given by a 
product of terms F; (£m, Xm) each associated with one of the factor nodes f; that is 
linked to node £m (excluding node fs), so that 


Crate Xen) = T Fier Xn) (8.68) 
lene(am)\fs 
where the product is taken over all neighbours of node £m except for node fs. Note 


that each of the factors F; (am, Xm) represents a subtree of the original graph of 
precisely the same kind as introduced in (8.62). Substituting (8.68) into (8.67), we 
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Figure 8.48 


Figure 8.49 


Illustration of the evaluation of the message sent by a fi 
variable node to an adjacent factor node. 


hi 
Pm Xni) 


then obtain 


Ham>fs (Em) = II 


lene(am)\fs 


II Uhin (Lm) (8.69) 


lene(a@m)\fs 


© Alem Xn] 


Xml 


where we have used the definition (8.64) of the messages passed from factor nodes to 
variable nodes. Thus to evaluate the message sent by a variable node to an adjacent 
factor node along the connecting link, we simply take the product of the incoming 
messages along all of the other links. Note that any variable node that has only 
two neighbours performs no computation but simply passes messages through un- 
changed. Also, we note that a variable node can send a message to a factor node 
once it has received incoming messages from all other neighbouring factor nodes. 
Recall that our goal is to calculate the marginal for variable node x, and that this 
marginal is given by the product of incoming messages along all of the links arriving 
at that node. Each of these messages can be computed recursively in terms of other 
messages. In order to start this recursion, we can view the node x as the root of the 
tree and begin at the leaf nodes. From the definition (8.69), we see that if a leaf node 
is a variable node, then the message that it sends along its one and only link is given 
by 
Hya) (8.70) 


as illustrated in Figure 8.49(a). Similarly, if the leaf node is a factor node, we see 
from (8.66) that the message sent should take the form 


Hf—~alz) = f(x) (8.71) 
The sum-product algorithm fig ape) =1 Uf>slz) = f(x) 


begins with messages sent —> —_> 
by the leaf nodes, which de- OS m—) 
pend on whether the leaf x f f 


node is (a) a variable node, 
or (b) a factor node. (a) (b) 
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as illustrated in Figure 8.49(b). 

At this point, it is worth pausing to summarize the particular version of the sum- 
product algorithm obtained so far for evaluating the marginal p(x). We start by 
viewing the variable node x as the root of the factor graph and initiating messages 
at the leaves of the graph using (8.70) and (8.71). The message passing steps (8.66) 
and (8.69) are then applied recursively until messages have been propagated along 
every link, and the root node has received messages from all of its neighbours. Each 
node can send a message towards the root once it has received messages from all 
of its other neighbours. Once the root node has received messages from all of its 
neighbours, the required marginal can be evaluated using (8.63). We shall illustrate 
this process shortly. 

To see that each node will always receive enough messages to be able to send out 
a message, we can use a simple inductive argument as follows. Clearly, for a graph 
comprising a variable root node connected directly to several factor leaf nodes, the 
algorithm trivially involves sending messages of the form (8.71) directly from the 
leaves to the root. Now imagine building up a general graph by adding nodes one at 
a time, and suppose that for some particular graph we have a valid algorithm. When 
one more (variable or factor) node is added, it can be connected only by a single 
link because the overall graph must remain a tree, and so the new node will be a leaf 
node. It therefore sends a message to the node to which it is linked, which in turn 
will therefore receive all the messages it requires in order to send its own message 
towards the root, and so again we have a valid algorithm, thereby completing the 
proof. 

Now suppose we wish to find the marginals for every variable node in the graph. 
This could be done by simply running the above algorithm afresh for each such node. 
However, this would be very wasteful as many of the required computations would 
be repeated. We can obtain a much more efficient procedure by ‘overlaying’ these 
multiple message passing algorithms to obtain the general sum-product algorithm 
as follows. Arbitrarily pick any (variable or factor) node and designate it as the 
root. Propagate messages from the leaves to the root as before. At this point, the 
root node will have received messages from all of its neighbours. It can therefore 
send out messages to all of its neighbours. These in turn will then have received 
messages from all of their neighbours and so can send out messages along the links 
going away from the root, and so on. In this way, messages are passed outwards 
from the root all the way to the leaves. By now, a message will have passed in 
both directions across every link in the graph, and every node will have received 
a message from all of its neighbours. Again a simple inductive argument can be 
used to verify the validity of this message passing protocol. Because every variable 
node will have received messages from all of its neighbours, we can readily calculate 
the marginal distribution for every variable in the graph. The number of messages 
that have to be computed is given by twice the number of links in the graph and 
so involves only twice the computation involved in finding a single marginal. By 
comparison, if we had run the sum-product algorithm separately for each node, the 
amount of computation would grow quadratically with the size of the graph. Note 
that this algorithm is in fact independent of which node was designated as the root, 
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Figure 8.50 The sum-product algorithm can be viewed 


Exercise 8.21 


purely in terms of messages sent out by factor 
nodes to other factor nodes. In this example, 
the outgoing message shown by the blue arrow 
is obtained by taking the product of all the in- 
coming messages shown by green arrows, mul- 
tiplying by the factor fs, and marginalizing over 
the variables x; and x2. 


and indeed the notion of one node having a special status was introduced only as a 
convenient way to explain the message passing protocol. 

Next suppose we wish to find the marginal distributions p(x,) associated with 
the sets of variables belonging to each of the factors. By a similar argument to that 
used above, it is easy to see that the marginal associated with a factor is given by the 
product of messages arriving at the factor node and the local factor at that node 


peny=f0%) | aoa) (8.72) 


i€ne(fs) 


in complete analogy with the marginals at the variable nodes. If the factors are 
parameterized functions and we wish to learn the values of the parameters using 
the EM algorithm, then these marginals are precisely the quantities we will need to 
calculate in the E step, as we shall see in detail when we discuss the hidden Markov 
model in Chapter 13. 

The message sent by a variable node to a factor node, as we have seen, is simply 
the product of the incoming messages on other links. We can if we wish view the 
sum-product algorithm in a slightly different form by eliminating messages from 
variable nodes to factor nodes and simply considering messages that are sent out by 
factor nodes. This is most easily seen by considering the example in Figure 8.50. 

So far, we have rather neglected the issue of normalization. If the factor graph 
was derived from a directed graph, then the joint distribution is already correctly nor- 
malized, and so the marginals obtained by the sum-product algorithm will similarly 
be normalized correctly. However, if we started from an undirected graph, then in 
general there will be an unknown normalization coefficient 1/Z. As with the simple 
chain example of Figure 8.38, this is easily handled by working with an unnormal- 
ized version p(x) of the joint distribution, where p(x) = p(x)/Z. We first run the 
sum-product algorithm to find the corresponding unnormalized marginals p(x; ). The 
coefficient 1/Z is then easily obtained by normalizing any one of these marginals, 
and this is computationally efficient because the normalization is done over a single 
variable rather than over the entire set of variables as would be required to normalize 


p(x) directly. 


At this point, it may be helpful to consider a simple example to illustrate the 
operation of the sum-product algorithm. Figure 8.51 shows a simple 4-node factor 
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Figure 8.51 A simple factor graph used to illustrate the 1 T2 T3 
sum-product algorithm. 


fa fo 


fe 


T4 


graph whose unnormalized joint distribution is given by 


P(X) = fal£1, £2) fo(2, £3) fe(£2, £4). (8.73) 


In order to apply the sum-product algorithm to this graph, let us designate node x3 
as the root, in which case there are two leaf nodes x, and x4. Starting with the leaf 
nodes, we then have the following sequence of six messages 


ijn) = > fa(wr,22) (8.75) 
May>f.(@4) = 1 (8.76) 
ipone = SFGate) (8.77) 
Mag fy(£2) = bf,—22(£2)Mf.02 (£2) (8.78) 
ipaa = > DOnna (8.79) 


The direction of flow of these messages is illustrated in Figure 8.52. Once this mes- 
sage propagation is complete, we can then propagate messages from the root node 
out to the leaf nodes, and these are given by 


Mas f,(%3) = 1 (8.80) 
frm (22) = J fo(z2, 23) (8.81) 
Hae fa (£2) = [fy—+ae (£2) Mf.—+09 (£2) (8.82) 
Mf, (£1) = S Jute E (8.83) 
Hzs—= fe (T2) = Hfa—>z(T2)HU fyrr (£2) (8.84) 
ipet = F Sate e (8.85) 


T2 
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T2 T3 Tı T2 T3 


— s — _ — —_ + s + + 


(a) (b) 


Figure 8.52 Flow of messages for the sum-product algorithm applied to the example graph in Figure 8.51. (a) 
From the leaf nodes xı and x4 towards the root node x3. (b) From the root node towards the leaf nodes. 


One message has now passed in each direction across each link, and we can now 
evaluate the marginals. As a simple check, let us verify that the marginal p(x2) is 
given by the correct expression. Using (8.63) and substituting for the messages using 
the above results, we have 


P(x2) = Hfa>r (£2) U fys (£2) Ufesa (x2) 


£ falera) £ Hew £ fena) 
SYY faler, 22) folwa, 23) fo(w2, 24) 
= dw) (8.86) 


t3 £a 


as required. 

So far, we have assumed that all of the variables in the graph are hidden. In most 
practical applications, a subset of the variables will be observed, and we wish to cal- 
culate posterior distributions conditioned on these observations. Observed nodes are 
easily handled within the sum-product algorithm as follows. Suppose we partition x 
into hidden variables h and observed variables v, and that the observed value of v 
is denoted V. Then we simply multiply the joint distribution p(x) by Į]; I (vi, vi), 
where I(v, 0) = 1 if v = Vand I(v,v) = 0 otherwise. This product corresponds 
to p(h,v = V) and hence is an unnormalized version of p(h|v = Vv). By run- 
ning the sum-product algorithm, we can efficiently calculate the posterior marginals 
p(h;|v = F) up to a normalization coefficient whose value can be found efficiently 
using a local computation. Any summations over variables in v then collapse into a 
single term. 

We have assumed throughout this section that we are dealing with discrete vari- 
ables. However, there is nothing specific to discrete variables either in the graphical 
framework or in the probabilistic construction of the sum-product algorithm. For 
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Example of a joint distribution over two binary variables for r=0 «r= 1 
which the maximum of the joint distribution occurs for dif- = 

. . y=0 0.3 0.4 
ferent variable values compared to the maxima of the two = 4 0.3 0.0 
marginals. y= f : 


continuous variables the summations are simply replaced by integrations. We shall 
give an example of the sum-product algorithm applied to a graph of linear-Gaussian 
variables when we consider linear dynamical systems. 


8.4.5 The max-sum algorithm 


The sum-product algorithm allows us to take a joint distribution p(x) expressed 
as a factor graph and efficiently find marginals over the component variables. Two 
other common tasks are to find a setting of the variables that has the largest prob- 
ability and to find the value of that probability. These can be addressed through a 
closely related algorithm called max-sum, which can be viewed as an application of 
dynamic programming in the context of graphical models (Cormen et al., 2001). 

A simple approach to finding latent variable values having high probability 
would be to run the sum-product algorithm to obtain the marginals p(x;) for ev- 
ery variable, and then, for each marginal in turn, to find the value x* that maximizes 
that marginal. However, this would give the set of values that are individually the 
most probable. In practice, we typically wish to find the set of values that jointly 
have the largest probability, in other words the vector x™** that maximizes the joint 
distribution, so that 


max 


x™* — arg max p(x) (8.87) 
for which the corresponding value of the joint probability will be given by 


pix") = max p(x). (8.88) 


In general, x™** is not the same as the set of x¥ values, as we can easily show using 
a simple example. Consider the joint distribution p(x, y) over two binary variables 
x,y € {0,1} given in Table 8.1. The joint distribution is maximized by setting x = 
1 and y = 0, corresponding the value 0.4. However, the marginal for p(x), obtained 
by summing over both values of y, is given by p(x = 0) = 0.6 and p(x = 1) = 0.4, 
and similarly the marginal for y is given by p(y = 0) = 0.7 and p(y = 1) = 0.3, 
and so the marginals are maximized by x = 0 and y = 0, which corresponds to a 
value of 0.3 for the joint distribution. In fact, it is not difficult to construct examples 
for which the set of individually most probable values has probability zero under the 
joint distribution. 

We therefore seek an efficient algorithm for finding the value of x that maxi- 
mizes the joint distribution p(x) and that will allow us to obtain the value of the 
joint distribution at its maximum. To address the second of these problems, we shall 
simply write out the max operator in terms of its components 


max p(x) = max... max p(x) (8.89) 
x Tı TM 
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where M is the total number of variables, and then substitute for p(x) using its 
expansion in terms of a product of factors. In deriving the sum-product algorithm, 
we made use of the distributive law (8.53) for multiplication. Here we make use of 
the analogous law for the max operator 


max(ab, ac) = amax(b, c) (8.90) 


which holds if a > 0 (as will always be the case for the factors in a graphical model). 
This allows us to exchange products with maximizations. 

Consider first the simple example of a chain of nodes described by (8.49). The 
evaluation of the probability maximum can be written as 


1 
max p(x) = z max::: max [Y1 2(21, £2) --- YN-1,N (N-11, £N )|] 
x £ı x 


1 
= z max [hiaten a) | . maxby-a.ylen-1ey)] , 
v1 TN 


As with the calculation of marginals, we see that exchanging the max and product 
operators results in a much more efficient computation, and one that is easily inter- 
preted in terms of messages passed from node x y backwards along the chain to node 
Ti- 

We can readily generalize this result to arbitrary tree-structured factor graphs 
by substituting the expression (8.59) for the factor graph expansion into (8.89) and 
again exchanging maximizations with products. The structure of this calculation is 
identical to that of the sum-product algorithm, and so we can simply translate those 
results into the present context. In particular, suppose that we designate a particular 
variable node as the ‘root’ of the graph. Then we start a set of messages propagating 
inwards from the leaves of the tree towards the root, with each node sending its 
message towards the root once it has received all incoming messages from its other 
neighbours. The final maximization is performed over the product of all messages 
arriving at the root node, and gives the maximum value for p(x). This could be called 
the max-product algorithm and is identical to the sum-product algorithm except that 
summations are replaced by maximizations. Note that at this stage, messages have 
been sent from leaves to the root, but not in the other direction. 

In practice, products of many small probabilities can lead to numerical under- 
flow problems, and so it is convenient to work with the logarithm of the joint distri- 
bution. The logarithm is a monotonic function, so that if a > b then lna > ln b, and 
hence the max operator and the logarithm function can be interchanged, so that 


ln (max p(x)) = max ln p(x). (8.91) 
The distributive property is preserved because 
max(a + b,a + c) = a + max(b, c). (8.92) 


Thus taking the logarithm simply has the effect of replacing the products in the 
max-product algorithm with sums, and so we obtain the max-sum algorithm. From 
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the results (8.66) and (8.69) derived earlier for the sum-product algorithm, we can 
readily write down the max-sum algorithm in terms of message passing simply by 
replacing ‘sum’ with ‘max’ and replacing products with sums of logarithms to give 


pple) = max [ln f(x, £1,..., £M) + ye lan flm) | (8.93) 
Li; TM 
me€ne(fs)\x 
te) = YS). Bae). (8.94) 
lEne(x)\ f 


The initial messages sent by the leaf nodes are obtained by analogy with (8.70) and 
(8.71) and are given by 


lsp) = 0 (8.95) 
ipa) = lnf(zx) (8.96) 


while at the root node the maximum probability can then be computed, by analogy 
with (8.63), using 


pe = max eee E (8.97) 
s€ne(x) 


So far, we have seen how to find the maximum of the joint distribution by prop- 
agating messages from the leaves to an arbitrarily chosen root node. The result will 
be the same irrespective of which node is chosen as the root. Now we turn to the 
second problem of finding the configuration of the variables for which the joint dis- 
tribution attains this maximum value. So far, we have sent messages from the leaves 
to the root. The process of evaluating (8.97) will also give the value «™** for the 
most probable value of the root node variable, defined by 


am —argmax| \) py,s0(2)| - (8.98) 


s€ne(x) 


At this point, we might be tempted simply to continue with the message passing al- 
gorithm and send messages from the root back out to the leaves, using (8.93) and 
(8.94), then apply (8.98) to all of the remaining variable nodes. However, because 
we are now maximizing rather than summing, it is possible that there may be mul- 
tiple configurations of x all of which give rise to the maximum value for p(x). In 
such cases, this strategy can fail because it is possible for the individual variable 
values obtained by maximizing the product of messages at each node to belong to 
different maximizing configurations, giving an overall configuration that no longer 
corresponds to a maximum. 

The problem can be resolved by adopting a rather different kind of message 
passing from the root node to the leaves. To see how this works, let us return once 
again to the simple chain example of N variables 71,...,2y each having K states, 
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Figure 8.53 A lattice, or trellis, diagram show- 
ing explicitly the K possible states (one per row 


EE 
of the diagram) for each of the variables x, inthe k — 1 go g 


chain model. In this illustration K = 3. The ar- 
row shows the direction of message passing in the 
max-product algorithm. For every state k of each 
variable x, (corresponding to column n of the dia- 
gram) the function ¢(xn) defines a unique state at 
the previous variable, indicated by the black lines. k = 2 g 
The two paths through the lattice correspond to 
configurations that give the global maximum of the 
joint probability distribution, and either of these 
can be found by tracing back along the black lines 
in the opposite direction to the arrow. 


n— 2 n—1 n n+1 


corresponding to the graph shown in Figure 8.38. Suppose we take node zy to be 
the root node. Then in the first phase, we propagate messages from the leaf node x, 
to the root node using 


Wat, Fo naan) m Ue sed Tn) 
Hhp-ayean(tn) = max [In fa-in(En-1, 2n) + fing 1-fr-tin(n) 
nm—1 


which follow from applying (8.94) and (8.93) to this particular graph. The initial 
message sent from the leaf node is simply 


play —fy.a(#1) = 0. (8.99) 


The most probable value for xy is then given by 


oy = arg max [Ufu i yey (tn)] - (8.100) 


N 


Now we need to determine the states of the previous variables that correspond to the 
same maximizing configuration. This can be done by keeping track of which values 
of the variables gave rise to the maximum state of each variable, in other words by 
storing quantities given by 


(Ln) = arg max [In Inataltais We) = ji, 2 nin Oa | . (8.101) 


Tn—1 


To understand better what is happening, it is helpful to represent the chain of vari- 
ables in terms of a lattice or trellis diagram as shown in Figure 8.53. Note that this 
is not a probabilistic graphical model because the nodes represent individual states 
of variables, while each variable corresponds to a column of such states in the di- 
agram. For each state of a given variable, there is a unique state of the previous 
variable that maximizes the probability (ties are broken either systematically or at 
random), corresponding to the function ¢(z,,) given by (8.101), and this is indicated 
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by the lines connecting the nodes. Once we know the most probable value of the fi- 
nal node xy, we can then simply follow the link back to find the most probable state 
of node x y_; and so on back to the initial node xı. This corresponds to propagating 
a message back down the chain using 

c= oe) (8.102) 
and is known as back-tracking. Note that there could be several values of x,,_; all 
of which give the maximum value in (8.101). Provided we chose one of these values 
when we do the back-tracking, we are assured of a globally consistent maximizing 
configuration. 

In Figure 8.53, we have indicated two paths, each of which we shall suppose 
corresponds to a global maximum of the joint probability distribution. If k = 2 
and k = 3 each represent possible values of x’°*, then starting from either state 
and tracing back along the black lines, which corresponds to iterating (8.102), we 
obtain a valid global maximum configuration. Note that if we had run a forward 
pass of max-sum message passing followed by a backward pass and then applied 
(8.98) at each node separately, we could end up selecting some states from one path 
and some from the other path, giving an overall configuration that is not a global 
maximizer. We see that it is necessary instead to keep track of the maximizing states 
during the forward pass using the functions ¢(«,,) and then use back-tracking to find 
a consistent solution. 

The extension to a general tree-structured factor graph should now be clear. If 
a message is sent from a factor node f to a variable node x, a maximization is 
performed over all other variable nodes x7,,..., xm that are neighbours of that fac- 
tor node, using (8.93). When we perform this maximization, we keep a record of 
which values of the variables x1,..., jg gave rise to the maximum. Then in the 
back-tracking step, having found «™**, we can then use these stored values to as- 
sign consistent maximizing states £P°*,..., £h- The max-sum algorithm, with 
back-tracking, gives an exact maximizing configuration for the variables provided 
the factor graph is a tree. An important application of this technique is for finding 
the most probable sequence of hidden states in a hidden Markov model, in which 
case it is known as the Viterbi algorithm. 

As with the sum-product algorithm, the inclusion of evidence in the form of 
observed variables is straightforward. The observed variables are clamped to their 
observed values, and the maximization is performed over the remaining hidden vari- 
ables. This can be shown formally by including identity functions for the observed 
variables into the factor functions, as we did for the sum-product algorithm. 

It is interesting to compare max-sum with the iterated conditional modes (ICM) 
algorithm described on page 389. Each step in ICM is computationally simpler be- 
cause the ‘messages’ that are passed from one node to the next comprise a single 
value consisting of the new state of the node for which the conditional distribution 
is maximized. The max-sum algorithm is more complex because the messages are 
functions of node variables x and hence comprise a set of K values for each pos- 
sible state of x. Unlike max-sum, however, ICM is not guaranteed to find a global 
maximum even for tree-structured graphs. 
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8.4.6 Exact inference in general graphs 


The sum-product and max-sum algorithms provide efficient and exact solutions 
to inference problems in tree-structured graphs. For many practical applications, 
however, we have to deal with graphs having loops. 

The message passing framework can be generalized to arbitrary graph topolo- 
gies, giving an exact inference procedure known as the junction tree algorithm (Lau- 
ritzen and Spiegelhalter, 1988; Jordan, 2007). Here we give a brief outline of the 
key steps involved. This is not intended to convey a detailed understanding of the 
algorithm, but rather to give a flavour of the various stages involved. If the starting 
point is a directed graph, it is first converted to an undirected graph by moraliza- 
tion, whereas if starting from an undirected graph this step is not required. Next the 
graph is triangulated, which involves finding chord-less cycles containing four or 
more nodes and adding extra links to eliminate such chord-less cycles. For instance, 
in the graph in Figure 8.36, the cycle A-C—B-—D-A is chord-less a link could be 
added between A and B or alternatively between C and D. Note that the joint dis- 
tribution for the resulting triangulated graph is still defined by a product of the same 
potential functions, but these are now considered to be functions over expanded sets 
of variables. Next the triangulated graph is used to construct a new tree-structured 
undirected graph called a join tree, whose nodes correspond to the maximal cliques 
of the triangulated graph, and whose links connect pairs of cliques that have vari- 
ables in common. The selection of which pairs of cliques to connect in this way is 
important and is done so as to give a maximal spanning tree defined as follows. Of 
all possible trees that link up the cliques, the one that is chosen is one for which the 
weight of the tree is largest, where the weight for a link is the number of nodes shared 
by the two cliques it connects, and the weight for the tree is the sum of the weights 
for the links. If the tree is condensed, so that any clique that is a subset of another 
clique is absorbed into the larger clique, this gives a junction tree. As a consequence 
of the triangulation step, the resulting tree satisfies the running intersection property, 
which means that if a variable is contained in two cliques, then it must also be con- 
tained in every clique on the path that connects them. This ensures that inference 
about variables will be consistent across the graph. Finally, a two-stage message 
passing algorithm, essentially equivalent to the sum-product algorithm, can now be 
applied to this junction tree in order to find marginals and conditionals. Although 
the junction tree algorithm sounds complicated, at its heart is the simple idea that 
we have used already of exploiting the factorization properties of the distribution to 
allow sums and products to be interchanged so that partial summations can be per- 
formed, thereby avoiding having to work directly with the joint distribution. The 
role of the junction tree is to provide a precise and efficient way to organize these 
computations. It is worth emphasizing that this is achieved using purely graphical 
operations! 

The junction tree is exact for arbitrary graphs and is efficient in the sense that 
for a given graph there does not in general exist a computationally cheaper approach. 
Unfortunately, the algorithm must work with the joint distributions within each node 
(each of which corresponds to a clique of the triangulated graph) and so the compu- 
tational cost of the algorithm is determined by the number of variables in the largest 
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clique and will grow exponentially with this number in the case of discrete variables. 
An important concept is the treewidth of a graph (Bodlaender, 1993), which is de- 
fined in terms of the number of variables in the largest clique. In fact, it is defined to 
be as one less than the size of the largest clique, to ensure that a tree has a treewidth 
of 1. Because there in general there can be multiple different junction trees that can 
be constructed from a given starting graph, the treewidth is defined by the junction 
tree for which the largest clique has the fewest variables. If the treewidth of the 
original graph is high, the junction tree algorithm becomes impractical. 


8.4.7 Loopy belief propagation 


For many problems of practical interest, it will not be feasible to use exact in- 
ference, and so we need to exploit effective approximation methods. An important 
class of such approximations, that can broadly be called variational methods, will be 
discussed in detail in Chapter 10. Complementing these deterministic approaches is 
a wide range of sampling methods, also called Monte Carlo methods, that are based 
on stochastic numerical sampling from distributions and that will be discussed at 
length in Chapter 11. 

Here we consider one simple approach to approximate inference in graphs with 
loops, which builds directly on the previous discussion of exact inference in trees. 
The idea is simply to apply the sum-product algorithm even though there is no guar- 
antee that it will yield good results. This approach is known as loopy belief propa- 
gation (Frey and MacKay, 1998) and is possible because the message passing rules 
(8.66) and (8.69) for the sum-product algorithm are purely local. However, because 
the graph now has cycles, information can flow many times around the graph. For 
some models, the algorithm will converge, whereas for others it will not. 

In order to apply this approach, we need to define a message passing schedule. 
Let us assume that one message is passed at a time on any given link and in any 
given direction. Each message sent from a node replaces any previous message sent 
in the same direction across the same link and will itself be a function only of the 
most recent messages received by that node at previous steps of the algorithm. 

We have seen that a message can only be sent across a link from a node when 
all other messages have been received by that node across its other links. Because 
there are loops in the graph, this raises the problem of how to initiate the message 
passing algorithm. To resolve this, we suppose that an initial message given by the 
unit function has been passed across every link in each direction. Every node is then 
in a position to send a message. 

There are now many possible ways to organize the message passing schedule. 
For example, the flooding schedule simultaneously passes a message across every 
link in both directions at each time step, whereas schedules that pass one message at 
a time are called serial schedules. 

Following Kschischnang et al. (2001), we will say that a (variable or factor) 
node a has a message pending on its link to a node b if node a has received any 
message on any of its other links since the last time it send a message to b. Thus, 
when a node receives a message on one of its links, this creates pending messages 
on all of its other links. Only pending messages need to be transmitted because 
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Exercise 8.29 


Exercises 
8.1 


8.2 


other messages would simply duplicate the previous message on the same link. For 
graphs that have a tree structure, any schedule that sends only pending messages 
will eventually terminate once a message has passed in each direction across every 
link. At this point, there are no pending messages, and the product of the received 
messages at every variable give the exact marginal. In graphs having loops, however, 
the algorithm may never terminate because there might always be pending messages, 
although in practice it is generally found to converge within a reasonable time for 
most applications. Once the algorithm has converged, or once it has been stopped 
if convergence is not observed, the (approximate) local marginals can be computed 
using the product of the most recently received incoming messages to each variable 
node or factor node on every link. 

In some applications, the loopy belief propagation algorithm can give poor re- 
sults, whereas in other applications it has proven to be very effective. In particular, 
state-of-the-art algorithms for decoding certain kinds of error-correcting codes are 
equivalent to loopy belief propagation (Gallager, 1963; Berrou et al., 1993; McEliece 
et al., 1998; MacKay and Neal, 1999; Frey, 1998). 


8.4.8 Learning the graph structure 


In our discussion of inference in graphical models, we have assumed that the 
structure of the graph is known and fixed. However, there is also interest in go- 
ing beyond the inference problem and learning the graph structure itself from data 
(Friedman and Koller, 2003). This requires that we define a space of possible struc- 
tures as well as a measure that can be used to score each structure. 

From a Bayesian viewpoint, we would ideally like to compute a posterior dis- 
tribution over graph structures and to make predictions by averaging with respect 
to this distribution. If we have a prior p(m) over graphs indexed by m, then the 
posterior distribution is given by 


p(m|D) x p(m)p(D|m) (8.103) 


where D is the observed data set. The model evidence p(D|m) then provides the 
score for each model. However, evaluation of the evidence involves marginalization 
over the latent variables and presents a challenging computational problem for many 
models. 

Exploring the space of structures can also be problematic. Because the number 
of different graph structures grows exponentially with the number of nodes, it is 
often necessary to resort to heuristics to find good candidates. 


(x) EY By marginalizing out the variables in order, show that the representation 
(8.5) for the joint distribution of a directed graph is correctly normalized, provided 
each of the conditional distributions is normalized. 


(x) FY Show that the property of there being no directed cycles in a directed 
graph follows from the statement that there exists an ordered numbering of the nodes 
such that for each node there are no links going to a lower-numbered node. 
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The joint distribution over three binary variables. p(a, b,c) 
0.192 
0.144 
0.048 
0.216 
0.192 
0.064 
0.048 
0.096 
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(xx) Consider three binary variables a,b,c € {0,1} having the joint distribution 
given in Table 8.2. Show by direct evaluation that this distribution has the property 
that a and b are marginally dependent, so that p(a,b) # p(a)p(b), but that they 
become independent when conditioned on c, so that p(a,b|c) = p(alc)p(b|c) for 
both c = 0 and c = 1. 


(xx) Evaluate the distributions p(a), p(b|c), and p(c|a) corresponding to the joint 
distribution given in Table 8.2. Hence show by direct evaluation that p(a, b,c) = 
p(a)p(cla)p(b|c). Draw the corresponding directed graph. 


œ FWY Draw a directed probabilistic graphical model corresponding to the 
relevance vector machine described by (7.79) and (7.80). 


(x) For the model shown in Figure 8.13, we have seen that the number of parameters 
required to specify the conditional distribution p(y|71,..., £m), where x; € {0,1}, 
could be reduced from 2™ to M + 1 by making use of the logistic sigmoid represen- 
tation (8.10). An alternative representation (Pearl, 1988) is given by 


M 
p(y = 1la1,...,0m) = 1- (1 — wo) | [0 - n)” (8.104) 


i=1 


where the parameters ju; represent the probabilities p(x; = 1), and jug is an additional 
parameters satisfying 0 < 4o < 1. The conditional distribution (8.104) is known as 
the noisy-OR. Show that this can be interpreted as a ‘soft’ (probabilistic) form of the 
logical OR function (i.e., the function that gives y = 1 whenever at least one of the 
xi = 1). Discuss the interpretation of uo. 


(xx) Using the recursion relations (8.15) and (8.16), show that the mean and covari- 
ance of the joint distribution for the graph shown in Figure 8.14 are given by (8.17) 
and (8.18), respectively. 


(x) EY Show that a L b,c| d implies a 1L b | d. 


(x) FY Using the d-separation criterion, show that the conditional distribution 
for a node x in a directed graph, conditioned on all of the nodes in the Markov 
blanket, is independent of the remaining variables in the graph. 


420 


8. GRAPHICAL MODELS 


Figure 8.54 


8.10 


8.11 


8.12 


8.13 


8.14 


8.15 


Example of a graphical model used to explore the con- a b 
ditional independence properties of the head-to-head 

path a—c—b when a descendant of c, namely the node 

d, is observed. 


(x) Consider the directed graph shown in Figure 8.54 in which none of the variables 
is observed. Show that a IL b | Ø. Suppose we now observe the variable d. Show 
that in general a 1 b | d. 


(xx) Consider the example of the car fuel system shown in Figure 8.21, and suppose 
that instead of observing the state of the fuel gauge G directly, the gauge is seen by 
the driver D who reports to us the reading on the gauge. This report is either that the 
gauge shows full D = 1 or that it shows empty D = 0. Our driver is a bit unreliable, 
as expressed through the following probabilities 


p(D=1|G=1) = 09 (8.105) 
p(D=0jG=0) = 09. (8.106) 


Suppose that the driver tells us that the fuel gauge shows empty, in other words 
that we observe D = 0. Evaluate the probability that the tank is empty given only 
this observation. Similarly, evaluate the corresponding probability given also the 
observation that the battery is flat, and note that this second probability is lower. 
Discuss the intuition behind this result, and relate the result to Figure 8.54. 


(x) FQ Show that there are 2("—1)/? distinct undirected graphs over a set of 
M distinct random variables. Draw the 8 possibilities for the case of M = 3. 


(x) Consider the use of iterated conditional modes (ICM) to minimize the energy 
function given by (8.42). Write down an expression for the difference in the values 
of the energy associated with the two states of a particular variable xj, with all other 
variables held fixed, and show that it depends only on quantities that are local to x; 
in the graph. 


(x) Consider a particular case of the energy function given by (8.42) in which the 
coefficients 9 = h = 0. Show that the most probable configuration of the latent 
variables is given by x; = y; for all 2. 


œx) [IY Show that the joint distribution p(£n—1, £n) for two neighbouring 
nodes in the graph shown in Figure 8.38 is given by an expression of the form (8.58). 
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(xx) Consider the inference problem of evaluating p(x,,|x) for the graph shown 
in Figure 8.38, for all nodes n € {1,..., N — 1}. Show that the message passing 
algorithm discussed in Section 8.4.1 can be used to solve this efficiently, and discuss 
which messages are modified and in what way. 


(xx) Consider a graph of the form shown in Figure 8.38 having N = 5 nodes, in 
which nodes x3 and z; are observed. Use d-separation to show that x IL z5 | z3. 
Show that if the message passing algorithm of Section 8.4.1 is applied to the evalu- 
ation of p(a2|x3, z5), the result will be independent of the value of x5. 


œ» FQ Show that a distribution represented by a directed tree can trivially 
be written as an equivalent distribution over the corresponding undirected tree. Also 
show that a distribution expressed as an undirected tree can, by suitable normaliza- 
tion of the clique potentials, be written as a directed tree. Calculate the number of 
distinct directed trees that can be constructed from a given undirected tree. 


(xx) Apply the sum-product algorithm derived in Section 8.4.4 to the chain-of- 
nodes model discussed in Section 8.4.1 and show that the results (8.54), (8.55), and 
(8.57) are recovered as a special case. 


œ FY Consider the message passing protocol for the sum-product algorithm on 
a tree-structured factor graph in which messages are first propagated from the leaves 
to an arbitrarily chosen root node and then from the root node out to the leaves. Use 
proof by induction to show that the messages can be passed in such an order that 
at every step, each node that must send a message has received all of the incoming 
messages necessary to construct its outgoing messages. 


œx» [QQ Show that the marginal distributions p(x,) over the sets of variables 
X, associated with each of the factors f,,(x;) in a factor graph can be found by first 
running the sum-product message passing algorithm and then evaluating the required 
marginals using (8.72). 


(x) Consider a tree-structured factor graph, in which a given subset of the variable 
nodes form a connected subgraph (i.e., any variable node of the subset is connected 
to at least one of the other variable nodes via a single factor node). Show how the 
sum-product algorithm can be used to compute the marginal distribution over that 
subset. 


œ» FY In Section 8.4.4, we showed that the marginal distribution p(x;) for a 
variable node x; in a factor graph is given by the product of the messages arriving at 
this node from neighbouring factor nodes in the form (8.63). Show that the marginal 
p(x;) can also be written as the product of the incoming message along any one of 
the links with the outgoing message along the same link. 


(xx) Show that the marginal distribution for the variables x, in a factor f,(x,) in 
a tree-structured factor graph, after running the sum-product message passing algo- 
rithm, can be written as the product of the message arriving at the factor node along 
all its links, times the local factor f(x,), in the form (8.72). 
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8.25 


8.26 


8.27 


8.28 


8.29 


(xx) In (8.86), we verified that the sum-product algorithm run on the graph in 
Figure 8.51 with node x3 designated as the root node gives the correct marginal for 
x2. Show that the correct marginals are obtained also for xı and x3. Similarly, show 
that the use of the result (8.72) after running the sum-product algorithm on this graph 
gives the correct joint distribution for £1, £2. 


(x) Consider a tree-structured factor graph over discrete variables, and suppose we 
wish to evaluate the joint distribution p(x,, £a) associated with two variables x, and 
x» that do not belong to a common factor. Define a procedure for using the sum- 
product algorithm to evaluate this joint distribution in which one of the variables is 
successively clamped to each of its allowed values. 


(xx) Consider two discrete variables x and y each having three possible states, for 
example x,y € {0,1,2}. Construct a joint distribution p(x, y) over these variables 
having the property that the value £ that maximizes the marginal p(x), along with 
the value y that maximizes the marginal p(y), together have probability zero under 
the joint distribution, so that p(z, Y) = 0. 


œ» FY The concept of a pending message in the sum-product algorithm for 
a factor graph was defined in Section 8.4.7. Show that if the graph has one or more 
cycles, there will always be at least one pending message irrespective of how long 
the algorithm runs. 


œ» EQ Show that if the sum-product algorithm is run on a factor graph with a 
tree structure (no loops), then after a finite number of messages have been sent, there 
will be no pending messages. 
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If we define a joint distribution over observed and latent variables, the correspond- 
ing distribution of the observed variables alone is obtained by marginalization. This 
allows relatively complex marginal distributions over observed variables to be ex- 
pressed in terms of more tractable joint distributions over the expanded space of 
observed and latent variables. The introduction of latent variables thereby allows 
complicated distributions to be formed from simpler components. In this chapter, 
we shall see that mixture distributions, such as the Gaussian mixture discussed in 
Section 2.3.9, can be interpreted in terms of discrete latent variables. Continuous 
latent variables will form the subject of Chapter 12. 

As well as providing a framework for building more complex probability dis- 
tributions, mixture models can also be used to cluster data. We therefore begin our 
discussion of mixture distributions by considering the problem of finding clusters 
in a set of data points, which we approach first using a nonprobabilistic technique 
called the K-means algorithm (Lloyd, 1982). Then we introduce the latent variable 
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9.1. 


view of mixture distributions in which the discrete latent variables can be interpreted 
as defining assignments of data points to specific components of the mixture. A gen- 
eral technique for finding maximum likelihood estimators in latent variable models 
is the expectation-maximization (EM) algorithm. We first of all use the Gaussian 
mixture distribution to motivate the EM algorithm in a fairly informal way, and then 
we give a more careful treatment based on the latent variable viewpoint. We shall 
see that the K-means algorithm corresponds to a particular nonprobabilistic limit of 
EM applied to mixtures of Gaussians. Finally, we discuss EM in some generality. 

Gaussian mixture models are widely used in data mining, pattern recognition, 
machine learning, and statistical analysis. In many applications, their parameters are 
determined by maximum likelihood, typically using the EM algorithm. However, as 
we shall see there are some significant limitations to the maximum likelihood ap- 
proach, and in Chapter 10 we shall show that an elegant Bayesian treatment can be 
given using the framework of variational inference. This requires little additional 
computation compared with EM, and it resolves the principal difficulties of maxi- 
mun likelihood while also allowing the number of components in the mixture to be 
inferred automatically from the data. 


K-means Clustering 


We begin by considering the problem of identifying groups, or clusters, of data points 
in a multidimensional space. Suppose we have a data set {x1, . . . , Xy } consisting 
of N observations of a random D-dimensional Euclidean variable x. Our goal is to 
partition the data set into some number K of clusters, where we shall suppose for 
the moment that the value of K is given. Intuitively, we might think of a cluster as 
comprising a group of data points whose inter-point distances are small compared 
with the distances to points outside of the cluster. We can formalize this notion by 
first introducing a set of D-dimensional vectors py, where k = 1,..., K, in which 
H; is a prototype associated with the k*" cluster. As we shall see shortly, we can 
think of the u, as representing the centres of the clusters. Our goal is then to find 
an assignment of data points to clusters, as well as a set of vectors {p} }, such that 
the sum of the squares of the distances of each data point to its closest vector py, is 
a minimum. 

It is convenient at this point to define some notation to describe the assignment 
of data points to clusters. For each data point Xn, we introduce a corresponding set 
of binary indicator variables ray € {0,1}, where k = 1,..., K describing which of 
the K clusters the data point x, is assigned to, so that if data point x,, is assigned to 
cluster k then rnk = 1, and rnj = 0 for j ¢ k. This is known as the 1-of-K coding 
scheme. We can then define an objective function, sometimes called a distortion 


measure, given by 
K 


N 
T= SOS rala eil (9.1) 


1 k=1 


which represents the sum of the squares of the distances of each data point to its 
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assigned vector p4;,. Our goal is to find values for the {rx} and the {j,1,,} so as to 
minimize J. We can do this through an iterative procedure in which each iteration 
involves two successive steps corresponding to successive optimizations with respect 
to the rng and the py. First we choose some initial values for the u}. Then in the first 
phase we minimize J with respect to the r,,,, keeping the jz, fixed. In the second 
phase we minimize J with respect to the up, keeping rnk fixed. This two-stage 
optimization is then repeated until convergence. We shall see that these two stages 
of updating r,,;, and updating u, correspond respectively to the E (expectation) and 
M (maximization) steps of the EM algorithm, and to emphasize this we shall use the 
terms E step and M step in the context of the k-means algorithm. 

Consider first the determination of the rng. Because J in (9.1) is a linear func- 
tion of rnp, this optimization can be performed easily to give a closed form solution. 
The terms involving different n are independent and so we can optimize for each 
n separately by choosing rnk to be 1 for whichever value of k gives the minimum 
value of ||x,, — y4;,||?. In other words, we simply assign the n‘ data point to the 
closest cluster centre. More formally, this can be expressed as 


(9.2) 


. a a — 2 
= 1 ifk= arg min; [Xn || 
0 otherwise. 
Now consider the optimization of the jz; with the rn held fixed. The objective 
function J is a quadratic function of wą, and it can be minimized by setting its 
derivative with respect to jz, to zero giving 


N 
2S" rak(Xn — by) = 0 (9.3) 
n=1 


which we can easily solve for u, to give 


— Xa TnkXn 


Hk = ak : 


The denominator in this expression is equal to the number of points assigned to 
cluster k, and so this result has a simple interpretation, namely set jz; equal to the 
mean of all of the data points x» assigned to cluster k. For this reason, the procedure 
is known as the k-means algorithm. 

The two phases of re-assigning data points to clusters and re-computing the clus- 
ter means are repeated in turn until there is no further change in the assignments (or 
until some maximum number of iterations is exceeded). Because each phase reduces 
the value of the objective function J, convergence of the algorithm is assured. How- 
ever, it may converge to a local rather than global minimum of J. The convergence 
properties of the K-means algorithm were studied by MacQueen (1967). 

The K-means algorithm is illustrated using the Old Faithful data set in Fig- 
ure 9.1. For the purposes of this example, we have made a linear re-scaling of the 
data, known as standardizing, such that each of the variables has zero mean and 
unit standard deviation. For this example, we have chosen K = 2, and so in this 


(9.4) 
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Figure 9.1 Illustration of the K-means algorithm using the re-scaled Old Faithful data set. (a) Green points 
denote the data set in a two-dimensional Euclidean space. The initial choices for centres u, and u, are shown 
by the red and blue crosses, respectively. (b) In the initial E step, each data point is assigned either to the red 
cluster or to the blue cluster, according to which cluster centre is nearer. This is equivalent to classifying the 
points according to which side of the perpendicular bisector of the two cluster centres, shown by the magenta 
line, they lie on. (c) In the subsequent M step, each cluster centre is re-computed to be the mean of the points 
assigned to the corresponding cluster. (d)—(i) show successive E and M steps through to final convergence of 
the algorithm. 


Figure 9.2 Plot of the cost function J given by 
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(9.1) after each E step (blue points) 1000} 
and M step (red points) of the K- 
means algorithm for the example 
shown in Figure 9.1. The algo- J 
rithm has converged after the third 
M step, and the final EM cycle pro- 
duces no changes in either the as- 500; 
signments or the prototype vectors. 


case, the assignment of each data point to the nearest cluster centre is equivalent to a 
classification of the data points according to which side they lie of the perpendicular 
bisector of the two cluster centres. A plot of the cost function J given by (9.1) for 
the Old Faithful example is shown in Figure 9.2. 

Note that we have deliberately chosen poor initial values for the cluster centres 
so that the algorithm takes several steps before convergence. In practice, a better 
initialization procedure would be to choose the cluster centres u, to be equal to a 
random subset of K data points. It is also worth noting that the k-means algorithm 
itself is often used to initialize the parameters in a Gaussian mixture model before 
applying the EM algorithm. 

A direct implementation of the K-means algorithm as discussed here can be 
relatively slow, because in each E step it is necessary to compute the Euclidean dis- 
tance between every prototype vector and every data point. Various schemes have 
been proposed for speeding up the k-means algorithm, some of which are based on 
precomputing a data structure such as a tree such that nearby points are in the same 
subtree (Ramasubramanian and Paliwal, 1990; Moore, 2000). Other approaches 
make use of the triangle inequality for distances, thereby avoiding unnecessary dis- 
tance calculations (Hodgson, 1998; Elkan, 2003). 

So far, we have considered a batch version of k-means in which the whole data 
set is used together to update the prototype vectors. We can also derive an on-line 
stochastic algorithm (MacQueen, 1967) by applying the Robbins-Monro procedure 
to the problem of finding the roots of the regression function given by the derivatives 
of J in (9.1) with respect to u. This leads to a sequential update in which, for each 
data point xn in turn, we update the nearest prototype pz, using 


HRY = BR + Im (Kn — MR) (9.5) 
where mn is the learning rate parameter, which is typically made to decrease mono- 
tonically as more data points are considered. 

The -means algorithm is based on the use of squared Euclidean distance as the 
measure of dissimilarity between a data point and a prototype vector. Not only does 
this limit the type of data variables that can be considered (it would be inappropriate 
for cases where some or all of the variables represent categorical labels for instance), 
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but it can also make the determination of the cluster means nonrobust to outliers. We 
can generalize the k-means algorithm by introducing a more general dissimilarity 
measure V(x, x’) between two vectors x and x’ and then minimizing the following 


distortion measure 
7 N K 
J =X Y rar V (Xn, Me) (9.6) 


n=1 k=1 


which gives the K-medoids algorithm. The E step again involves, for given cluster 
prototypes j2,, assigning each data point to the cluster for which the dissimilarity to 
the corresponding prototype is smallest. The computational cost of this is O( KN), 
as is the case for the standard K-means algorithm. For a general choice of dissimi- 
larity measure, the M step is potentially more complex than for K’-means, and so it 
is common to restrict each cluster prototype to be equal to one of the data vectors as- 
signed to that cluster, as this allows the algorithm to be implemented for any choice 
of dissimilarity measure V(-,-) so long as it can be readily evaluated. Thus the M 
step involves, for each cluster k, a discrete search over the N;, points assigned to that 
cluster, which requires O(N?) evaluations of V(-,-). 

One notable feature of the K-means algorithm is that at each iteration, every 
data point is assigned uniquely to one, and only one, of the clusters. Whereas some 
data points will be much closer to a particular centre jz, than to any other centre, 
there may be other data points that lie roughly midway between cluster centres. In 
the latter case, it is not clear that the hard assignment to the nearest cluster is the 
most appropriate. We shall see in the next section that by adopting a probabilistic 
approach, we obtain ‘soft’ assignments of data points to clusters in a way that reflects 
the level of uncertainty over the most appropriate assignment. This probabilistic 
formulation brings with it numerous benefits. 


9.1.1 Image segmentation and compression 


As an illustration of the application of the K-means algorithm, we consider 
the related problems of image segmentation and image compression. The goal of 
segmentation is to partition an image into regions each of which has a reasonably 
homogeneous visual appearance or which corresponds to objects or parts of objects 
(Forsyth and Ponce, 2003). Each pixel in an image is a point in a 3-dimensional space 
comprising the intensities of the red, blue, and green channels, and our segmentation 
algorithm simply treats each pixel in the image as a separate data point. Note that 
strictly this space is not Euclidean because the channel intensities are bounded by 
the interval [0, 1]. Nevertheless, we can apply the K-means algorithm without diffi- 
culty. We illustrate the result of running K’-means to convergence, for any particular 
value of K, by re-drawing the image replacing each pixel vector with the {R,G, B} 
intensity triplet given by the centre w, to which that pixel has been assigned. Results 
for various values of K are shown in Figure 9.3. We see that for a given value of K, 
the algorithm is representing the image using a palette of only K colours. It should 
be emphasized that this use of k-means is not a particularly sophisticated approach 
to image segmentation, not least because it takes no account of the spatial proximity 
of different pixels. The image segmentation problem is in general extremely difficult 
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Original image 


Two examples of the application of the k-means clustering algorithm to image segmentation show- 
ing the initial images together with their AK-means segmentations obtained using various values of K. This 
also illustrates of the use of vector quantization for data compression, in which smaller values of K give higher 
compression at the expense of poorer image quality. 


and remains the subject of active research and is introduced here simply to illustrate 
the behaviour of the K-means algorithm. 

We can also use the result of a clustering algorithm to perform data compres- 
sion. It is important to distinguish between lossless data compression, in which 
the goal is to be able to reconstruct the original data exactly from the compressed 
representation, and lossy data compression, in which we accept some errors in the 
reconstruction in return for higher levels of compression than can be achieved in the 
lossless case. We can apply the K-means algorithm to the problem of lossy data 
compression as follows. For each of the N data points, we store only the identity 
k of the cluster to which it is assigned. We also store the values of the K clus- 
ter centres 4y, which typically requires significantly less data, provided we choose 
K < N. Each data point is then approximated by its nearest centre up. New data 
points can similarly be compressed by first finding the nearest 4, and then storing 
the label k instead of the original data vector. This framework is often called vector 
quantization, and the vectors u, are called code-book vectors. 
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The image segmentation problem discussed above also provides an illustration 
of the use of clustering for data compression. Suppose the original image has N 
pixels comprising {R, G, B} values each of which is stored with 8 bits of precision. 
Then to transmit the whole image directly would cost 24N bits. Now suppose we 
first run K-means on the image data, and then instead of transmitting the original 
pixel intensity vectors we transmit the identity of the nearest vector up. Because 
there are K such vectors, this requires log, K bits per pixel. We must also transmit 
the K code book vectors wą, which requires 24K bits, and so the total number of 
bits required to transmit the image is 24K + N log, K (rounding up to the nearest 
integer). The original image shown in Figure 9.3 has 240 x 180 = 43, 200 pixels 
and so requires 24 x 43, 200 = 1, 036, 800 bits to transmit directly. By comparison, 
the compressed images require 43, 248 bits (K = 2), 86,472 bits (K = 3), and 
173, 040 bits (K = 10), respectively, to transmit. These represent compression ratios 
compared to the original image of 4.2%, 8.3%, and 16.7%, respectively. We see that 
there is a trade-off between degree of compression and image quality. Note that our 
aim in this example is to illustrate the k-means algorithm. If we had been aiming to 
produce a good image compressor, then it would be more fruitful to consider small 
blocks of adjacent pixels, for instance 5 x 5, and thereby exploit the correlations that 
exist in natural images between nearby pixels. 


Mixtures of Gaussians 


In Section 2.3.9 we motivated the Gaussian mixture model as a simple linear super- 
position of Gaussian components, aimed at providing a richer class of density mod- 
els than the single Gaussian. We now turn to a formulation of Gaussian mixtures in 
terms of discrete latent variables. This will provide us with a deeper insight into this 
important distribution, and will also serve to motivate the expectation-maximization 
algorithm. 

Recall from (2.188) that the Gaussian mixture distribution can be written as a 
linear superposition of Gaussians in the form 


K 
p(x) = So tN (Xba, Xz). (9.7) 


k=1 


Let us introduce a K-dimensional binary random variable z having a 1-of-K repre- 
sentation in which a particular element z% is equal to 1 and all other elements are 
equal to 0. The values of zx therefore satisfy zx € {0,1} and 5 , Zk = 1, and we 
see that there are K possible states for the vector z according to which element is 
nonzero. We shall define the joint distribution p(x, z) in terms of a marginal dis- 
tribution p(z) and a conditional distribution p(x|z), corresponding to the graphical 
model in Figure 9.4. The marginal distribution over z is specified in terms of the 
mixing coefficients mg, such that 
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Figure 9.4 Graphical representation of a mixture model, in which Z 
the joint distribution is expressed in the form p(x,z) = 
p(z)p(x|z). 
x 
where the parameters {7} must satisfy 
O<m<l (9.8) 
together with 
K 
Som =1 (9.9) 
k=1 


Exercise 9.3 


in order to be valid probabilities. Because z uses a 1-of-K representation, we can 
also write this distribution in the form 


K 
p(z) = II ee (9.10) 
k=1 


Similarly, the conditional distribution of x given a particular value for z is a Gaussian 
P(x|2% = 1) = N(x| My, Be) 


which can also be written in the form 


K 


p(x|z) = [Vel De): (9.11) 


k=1 


The joint distribution is given by p(z)p(x|z), and the marginal distribution of x is 
then obtained by summing the joint distribution over all possible states of z to give 


K 
p(x) = X > p(z)p(x|z) = So tM (x|beys Dx) (9.12) 


k=1 


where we have made use of (9.10) and (9.11). Thus the marginal distribution of x is 
a Gaussian mixture of the form (9.7). If we have several observations X1,..., XN, 
then, because we have represented the marginal distribution in the form p(x) = 
$ P(x, z), it follows that for every observed data point x, there is a corresponding 
latent variable z,,. 

We have therefore found an equivalent formulation of the Gaussian mixture in- 
volving an explicit latent variable. It might seem that we have not gained much 
by doing so. However, we are now able to work with the joint distribution p(x, z) 
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instead of the marginal distribution p(x), and this will lead to significant simplifica- 
tions, most notably through the introduction of the expectation-maximization (EM) 
algorithm. 

Another quantity that will play an important role is the conditional probability 
of z given x. We shall use 7(z;,) to denote p(z = 1|x), whose value can be found 
using Bayes’ theorem 


plk = 1)p(x|ze = 1) 
K 
> pl = 1)p(x|z; = 1) 


_ TEN (x| My, Ee) (9.13) 


K 
`> TiN (x| uj, xj) 
j=l 


Y(zk) = p(zk = 1x) = 


We shall view 7y as the prior probability of zx = 1, and the quantity 7(z;) as the 
corresponding posterior probability once we have observed x. As we shall see later, 
y(zķ) can also be viewed as the responsibility that component k takes for “explain- 
ing’ the observation x. 

We can use the technique of ancestral sampling to generate random samples 
distributed according to the Gaussian mixture model. To do this, we first generate a 
value for z, which we denote z, from the marginal distribution p(z) and then generate 
a value for x from the conditional distribution p(x|z). Techniques for sampling from 
standard distributions are discussed in Chapter 11. We can depict samples from the 
joint distribution p(x,z) by plotting points at the corresponding values of x and 
then colouring them according to the value of z, in other words according to which 
Gaussian component was responsible for generating them, as shown in Figure 9.5(a). 
Similarly samples from the marginal distribution p(x) are obtained by taking the 
samples from the joint distribution and ignoring the values of z. These are illustrated 
in Figure 9.5(b) by plotting the x values without any coloured labels. 

We can also use this synthetic data set to illustrate the ‘responsibilities’ by eval- 
uating, for every data point, the posterior probability for each component in the 
mixture distribution from which this data set was generated. In particular, we can 
represent the value of the responsibilities 7(z,,,) associated with data point x„ by 
plotting the corresponding point using proportions of red, blue, and green ink given 
by ¥(Znx) for k = 1, 2,3, respectively, as shown in Figure 9.5(c). So, for instance, 
a data point for which 7(2n1) = 1 will be coloured red, whereas one for which 
Y(Zn2) = Y(Zn3) = 0.5 will be coloured with equal proportions of blue and green 
ink and so will appear cyan. This should be compared with Figure 9.5(a) in which 
the data points were labelled using the true identity of the component from which 
they were generated. 


9.2.1 Maximum likelihood 


Suppose we have a data set of observations {x1,..., Xy }, and we wish to model 
this data using a mixture of Gaussians. We can represent this data set as an N x D 
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Figure 9.5 Example of 500 points drawn from the mixture of 3 Gaussians shown in Figure 2.23. (a) Samples 
from the joint distribution p(z)p(x|z) in which the three states of z, corresponding to the three components of the 
mixture, are depicted in red, green, and blue, and (b) the corresponding samples from the marginal distribution 
p(x), which is obtained by simply ignoring the values of z and just plotting the x values. The data set in (a) is 
said to be complete, whereas that in (b) is incomplete. (c) The same samples in which the colours represent the 
value of the responsibilities y(znx) associated with data point xn, obtained by plotting the corresponding point 
using proportions of red, blue, and green ink given by 7(znx) for k = 1, 2,3, respectively 


Figure 9.6 


matrix X in which the nt row is given by xT. Similarly, the corresponding latent 
variables will be denoted by an N x K matrix Z with rows zT. If we assume that 
the data points are drawn independently from the distribution, then we can express 
the Gaussian mixture model for this 1.i.d. data set using the graphical representation 
shown in Figure 9.6. From (9.7) the log of the likelihood function is given by 


N K 
In p(X|7, u, ©) = Soin} Sm Monn Bs) f (9.14) 
k=1 


n=1 


Before discussing how to maximize this function, it is worth emphasizing that 
there is a significant problem associated with the maximum likelihood framework 
applied to Gaussian mixture models, due to the presence of singularities. For sim- 
plicity, consider a Gaussian mixture whose components have covariance matrices 
given by Xp = o7I, where I is the unit matrix, although the conclusions will hold 
for general covariance matrices. Suppose that one of the components of the mixture 


model, let us say the j*" component, has its mean H; exactly equal to one of the data 


Graphical representation of a Gaussian mixture model 
for a set of N i.i.d. data points {xn }, with corresponding 
latent points {zn}, where n = 1,..., N. Gi () 


om G7 ee 
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Illustration of how singularities in the 
likelihood function arise with mixtures 

of Gaussians. This should be com- p(x) 
pared with the case of a single Gaus- 

sian shown in Figure 1.14 for which no 
singularities arise. 


points so that u; = Xn for some value of n. This data point will then contribute a 
term in the likelihood function of the form 


1 i 
(21)? o5 


If we consider the limit 7; — 0, then we see that this term goes to infinity and 
so the log likelihood function will also go to infinity. Thus the maximization of 
the log likelihood function is not a well posed problem because such singularities 
will always be present and will occur whenever one of the Gaussian components 
‘collapses’ onto a specific data point. Recall that this problem did not arise in the 
case of a single Gaussian distribution. To understand the difference, note that if a 
single Gaussian collapses onto a data point it will contribute multiplicative factors 
to the likelihood function arising from the other data points and these factors will go 
to zero exponentially fast, giving an overall likelihood that goes to zero rather than 
infinity. However, once we have (at least) two components in the mixture, one of 
the components can have a finite variance and therefore assign finite probability to 
all of the data points while the other component can shrink onto one specific data 
point and thereby contribute an ever increasing additive value to the log likelihood. 
This is illustrated in Figure 9.7. These singularities provide another example of the 
severe over-fitting that can occur in a maximum likelihood approach. We shall see 
that this difficulty does not occur if we adopt a Bayesian approach. For the moment, 
however, we simply note that in applying maximum likelihood to Gaussian mixture 
models we must take steps to avoid finding such pathological solutions and instead 
seek local maxima of the likelihood function that are well behaved. We can hope to 
avoid the singularities by using suitable heuristics, for instance by detecting when a 
Gaussian component is collapsing and resetting its mean to a randomly chosen value 
while also resetting its covariance to some large value, and then continuing with the 
optimization. 

A further issue in finding maximum likelihood solutions arises from the fact 
that for any given maximum likelihood solution, a K-component mixture will have 
a total of K! equivalent solutions corresponding to the K! ways of assigning K 
sets of parameters to K components. In other words, for any given (nondegenerate) 
point in the space of parameter values there will be a further K! — 1 additional points 
all of which give rise to exactly the same distribution. This problem is known as 


N (Xnl|Xn, a+) = (9.15) 
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identifiability (Casella and Berger, 2002) and is an important issue when we wish to 
interpret the parameter values discovered by a model. Identifiability will also arise 
when we discuss models having continuous latent variables in Chapter 12. However, 
for the purposes of finding a good density model, it is irrelevant because any of the 
equivalent solutions is as good as any other. 

Maximizing the log likelihood function (9.14) for a Gaussian mixture model 
turns out to be a more complex problem than for the case of a single Gaussian. The 
difficulty arises from the presence of the summation over k that appears inside the 
logarithm in (9.14), so that the logarithm function no longer acts directly on the 
Gaussian. If we set the derivatives of the log likelihood to zero, we will no longer 
obtain a closed form solution, as we shall see shortly. 

One approach is to apply gradient-based optimization techniques (Fletcher, 1987; 
Nocedal and Wright, 1999; Bishop and Nabney, 2008). Although gradient-based 
techniques are feasible, and indeed will play an important role when we discuss 
mixture density networks in Chapter 5, we now consider an alternative approach 
known as the EM algorithm which has broad applicability and which will lay the 
foundations for a discussion of variational inference techniques in Chapter 10. 


9.2.2 EM for Gaussian mixtures 


An elegant and powerful method for finding maximum likelihood solutions for 
models with latent variables is called the expectation-maximization algorithm, or EM 
algorithm (Dempster et al., 1977; McLachlan and Krishnan, 1997). Later we shall 
give a general treatment of EM, and we shall also show how EM can be generalized 
to obtain the variational inference framework. Initially, we shall motivate the EM 
algorithm by giving a relatively informal treatment in the context of the Gaussian 
mixture model. We emphasize, however, that EM has broad applicability, and indeed 
it will be encountered in the context of a variety of different models in this book. 

Let us begin by writing down the conditions that must be satisfied at a maximum 
of the likelihood function. Setting the derivatives of In p(X|7r, u, ©) in (9.14) with 
respect to the means py of the Gaussian components to zero, we obtain 


TN ( (Xnl Hk, Be) 
Egl(Xn — Hg) (9.16) 
-y TiN (Xn|Mj, Xj) Kl i 
1E ne ene AS 


¥(Znk) 
where we have made use of the form (2.43) for the Gaussian distribution. Note that 
the posterior probabilities, or responsibilities, given by (9.13) appear naturally on 
the right-hand side. Multiplying by bie (which we assume to be nonsingular) and 
rearranging we obtain 


N 
1 
SA nk)Xn 9.17 
Me = dW k)X (9.17) 
where we have defined a 
Ne = J Wn). (9.18) 
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We can interpret Nj, as the effective number of points assigned to cluster k. Note 
carefully the form of this solution. We see that the mean p, for the k*" Gaussian 
component is obtained by taking a weighted mean of all of the points in the data set, 
in which the weighting factor for data point x,, is given by the posterior probability 
Y(Znk) that component k was responsible for generating Xn. 

If we set the derivative of ln p(X|7r, ps, X) with respect to X, to zero, and follow 
a similar line of reasoning, making use of the result for the maximum likelihood 
solution for the covariance matrix of a single Gaussian, we obtain 


N 
1 
X, = N, N Wear) En — Hy) (Xn — Mp)” (9.19) 
n=1 


which has the same form as the corresponding result for a single Gaussian fitted to 
the data set, but again with each data point weighted by the corresponding poste- 
rior probability and with the denominator given by the effective number of points 
associated with the corresponding component. 

Finally, we maximize In p(X|7r, u, ©) with respect to the mixing coefficients 
Tk. Here we must take account of the constraint (9.9), which requires the mixing 
coefficients to sum to one. This can be achieved using a Lagrange multiplier and 
maximizing the following quantity 


K 
In p(X|7, u, £) + A (>: Tk — 7 (9.20) 
k=1 


which gives 


N 
N (Xn| Hk, Uk) 
0= HÀ 9.21 

> ar TiN (Xn | Hj, Dy) i } 


where again we see the appearance of the responsibilities. If we now multiply both 
sides by 7; and sum over k making use of the constraint (9.9), we find ÀA = —N. 
Using this to eliminate À and rearranging we obtain 


OoN 


Tk (9.22) 
so that the mixing coefficient for the k'* component is given by the average respon- 
sibility which that component takes for explaining the data points. 

It is worth emphasizing that the results (9.17), (9.19), and (9.22) do not con- 
stitute a closed-form solution for the parameters of the mixture model because the 
responsibilities y(z,;) depend on those parameters in a complex way through (9.13). 
However, these results do suggest a simple iterative scheme for finding a solution to 
the maximum likelihood problem, which as we shall see turns out to be an instance 
of the EM algorithm for the particular case of the Gaussian mixture model. We 
first choose some initial values for the means, covariances, and mixing coefficients. 
Then we alternate between the following two updates that we shall call the E step 
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Figure 9.8 


Illustration of the EM algorithm using the Old Faithful set as used for the illustration of the K-means 


algorithm in Figure 9.1. See the text for details. 
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and the M step, for reasons that will become apparent shortly. In the expectation 
step, or E step, we use the current values for the parameters to evaluate the posterior 
probabilities, or responsibilities, given by (9.13). We then use these probabilities in 
the maximization step, or M step, to re-estimate the means, covariances, and mix- 
ing coefficients using the results (9.17), (9.19), and (9.22). Note that in so doing 
we first evaluate the new means using (9.17) and then use these new values to find 
the covariances using (9.19), in keeping with the corresponding result for a single 
Gaussian distribution. We shall show that each update to the parameters resulting 
from an E step followed by an M step is guaranteed to increase the log likelihood 
function. In practice, the algorithm is deemed to have converged when the change 
in the log likelihood function, or alternatively in the parameters, falls below some 
threshold. We illustrate the EM algorithm for a mixture of two Gaussians applied to 
the rescaled Old Faithful data set in Figure 9.8. Here a mixture of two Gaussians 
is used, with centres initialized using the same values as for the K-means algorithm 
in Figure 9.1, and with precision matrices initialized to be proportional to the unit 
matrix. Plot (a) shows the data points in green, together with the initial configura- 
tion of the mixture model in which the one standard-deviation contours for the two 
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Gaussian components are shown as blue and red circles. Plot (b) shows the result 
of the initial E step, in which each data point is depicted using a proportion of blue 
ink equal to the posterior probability of having been generated from the blue com- 
ponent, and a corresponding proportion of red ink given by the posterior probability 
of having been generated by the red component. Thus, points that have a significant 
probability for belonging to either cluster appear purple. The situation after the first 
M step is shown in plot (c), in which the mean of the blue Gaussian has moved to 
the mean of the data set, weighted by the probabilities of each data point belonging 
to the blue cluster, in other words it has moved to the centre of mass of the blue ink. 
Similarly, the covariance of the blue Gaussian is set equal to the covariance of the 
blue ink. Analogous results hold for the red component. Plots (d), (e), and (f) show 
the results after 2, 5, and 20 complete cycles of EM, respectively. In plot (f) the 
algorithm is close to convergence. 

Note that the EM algorithm takes many more iterations to reach (approximate) 
convergence compared with the k-means algorithm, and that each cycle requires 
significantly more computation. It is therefore common to run the -means algo- 
rithm in order to find a suitable initialization for a Gaussian mixture model that is 
subsequently adapted using EM. The covariance matrices can conveniently be ini- 
tialized to the sample covariances of the clusters found by the K-means algorithm, 
and the mixing coefficients can be set to the fractions of data points assigned to the 
respective clusters. As with gradient-based approaches for maximizing the log like- 
lihood, techniques must be employed to avoid singularities of the likelihood function 
in which a Gaussian component collapses onto a particular data point. It should be 
emphasized that there will generally be multiple local maxima of the log likelihood 
function, and that EM is not guaranteed to find the largest of these maxima. Because 
the EM algorithm for Gaussian mixtures plays such an important role, we summarize 
it below. 


EM for Gaussian Mixtures 


Given a Gaussian mixture model, the goal is to maximize the likelihood function 
with respect to the parameters (comprising the means and covariances of the 
components and the mixing coefficients). 


1. Initialize the means pg, covariances Xp and mixing coefficients m, and 
evaluate the initial value of the log likelihood. 


2. Estep. Evaluate the responsibilities using the current parameter values 


AN Soe) 


San (xnl; © j) 


Y(Znk) = (9.23) 
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3. M step. Re-estimate the parameters using the current responsibilities 


N 
new roa 1 
ie = ae >, Y(2nk)Xn (9.24) 
1 N 
e =w N Wenk) Xn — HRY) (xn — aR) (9.25) 
n=1 
Ni, 
ew = at (9.26) 
where 
N 
n=1 


4. Evaluate the log likelihood 


N K 
In p(X|p, £, 3) = Ynf So aNu z] (9.28) 
k=1 


n=1 


and check for convergence of either the parameters or the log likelihood. If 
the convergence criterion is not satisfied return to step 2. 


9.3. An Alternative View of EM 


In this section, we present a complementary view of the EM algorithm that recog- 
nizes the key role played by latent variables. We discuss this approach first of all 
in an abstract setting, and then for illustration we consider once again the case of 
Gaussian mixtures. 

The goal of the EM algorithm is to find maximum likelihood solutions for mod- 
els having latent variables. We denote the set of all observed data by X, in which the 
n* row represents x7, and similarly we denote the set of all latent variables by Z, 
with a corresponding row zT. The set of all model parameters is denoted by 0, and 
so the log likelihood function is given by 


In p(X|@) = In [Za zo} . (9.29) 


Note that our discussion will apply equally well to continuous latent variables simply 
by replacing the sum over Z with an integral. 

A key observation is that the summation over the latent variables appears inside 
the logarithm. Even if the joint distribution p(X, Z|@) belongs to the exponential 
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family, the marginal distribution p(X|@) typically does not as a result of this sum- 
mation. The presence of the sum prevents the logarithm from acting directly on the 
joint distribution, resulting in complicated expressions for the maximum likelihood 
solution. 

Now suppose that, for each observation in X, we were told the corresponding 
value of the latent variable Z. We shall call {X, Z} the complete data set, and we 
shall refer to the actual observed data X as incomplete, as illustrated in Figure 9.5. 
The likelihood function for the complete data set simply takes the form In p(X, Z|@), 
and we shall suppose that maximization of this complete-data log likelihood function 
is straightforward. 

In practice, however, we are not given the complete data set {X, Z}, but only 
the incomplete data X. Our state of knowledge of the values of the latent variables 
in Z is given only by the posterior distribution p(Z|X, 0). Because we cannot use 
the complete-data log likelihood, we consider instead its expected value under the 
posterior distribution of the latent variable, which corresponds (as we shall see) to the 
E step of the EM algorithm. In the subsequent M step, we maximize this expectation. 
If the current estimate for the parameters is denoted 9°", then a pair of successive 
E and M steps gives rise to a revised estimate 8"°”. The algorithm is initialized by 
choosing some starting value for the parameters 09. The use of the expectation may 
seem somewhat arbitrary. However, we shall see the motivation for this choice when 
we give a deeper treatment of EM in Section 9.4. 

In the E step, we use the current parameter values 0° to find the posterior 
distribution of the latent variables given by p(Z|X, 0°"). We then use this posterior 
distribution to find the expectation of the complete-data log likelihood evaluated for 
some general parameter value @. This expectation, denoted Q(0, 0°"), is given by 


Q(0,0°") = X` p(Z|X, 0%) In p(X, ZI6). (9.30) 
Z 


In the M step, we determine the revised parameter estimate 0"°” by maximizing this 
function 
0™™ = arg max O(6, 0°"). (9.31) 
6 


Note that in the definition of Q(0,0°'"), the logarithm acts directly on the joint 
distribution p(X, Z|@), and so the corresponding M-step maximization will, by sup- 
position, be tractable. 

The general EM algorithm is summarized below. It has the property, as we shall 
show later, that each cycle of EM will increase the incomplete-data log likelihood 
(unless it is already at a local maximum). 


The General EM Algorithm 


Given a joint distribution p(X, Z|@) over observed variables X and latent vari- 
ables Z, governed by parameters 0, the goal is to maximize the likelihood func- 
tion p(X|@) with respect to 0. 


1. Choose an initial setting for the parameters 0°". 
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2. Estep Evaluate p(Z|X, 0°"). 


3. M step Evaluate 0™°™ given by 
6" = arg max Q(0, 0°"") (9.32) 
0 


where 
Q(0,0°") = X` p(Z|X, 0") In p(X, Z]0). (9.33) 


Z 


4. Check for convergence of either the log likelihood or the parameter values. 
If the convergence criterion is not satisfied, then let 


gold — grew (9.34) 


and return to step 2. 


The EM algorithm can also be used to find MAP (maximum posterior) solutions 
for models in which a prior p(@) is defined over the parameters. In this case the E 
step remains the same as in the maximum likelihood case, whereas in the M step the 
quantity to be maximized is given by Q(0, 0°") + In p(0). Suitable choices for the 
prior will remove the singularities of the kind illustrated in Figure 9.7. 

Here we have considered the use of the EM algorithm to maximize a likelihood 
function when there are discrete latent variables. However, it can also be applied 
when the unobserved variables correspond to missing values in the data set. The 
distribution of the observed values is obtained by taking the joint distribution of all 
the variables and then marginalizing over the missing ones. EM can then be used 
to maximize the corresponding likelihood function. We shall show an example of 
the application of this technique in the context of principal component analysis in 
Figure 12.11. This will be a valid procedure if the data values are missing at random, 
meaning that the mechanism causing values to be missing does not depend on the 
unobserved values. In many situations this will not be the case, for instance if a 
sensor fails to return a value whenever the quantity it is measuring exceeds some 
threshold. 


9.3.1 Gaussian mixtures revisited 


We now consider the application of this latent variable view of EM to the spe- 
cific case of a Gaussian mixture model. Recall that our goal is to maximize the log 
likelihood function (9.14), which is computed using the observed data set X, and we 
saw that this was more difficult than for the case of a single Gaussian distribution 
due to the presence of the summation over k that occurs inside the logarithm. Sup- 
pose then that in addition to the observed data set X, we were also given the values 
of the corresponding discrete variables Z. Recall that Figure 9.5(a) shows a ‘com- 
plete’ data set (i.e., one that includes labels showing which component generated 
each data point) while Figure 9.5(b) shows the corresponding ‘incomplete’ data set. 
The graphical model for the complete data is shown in Figure 9.9. 
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Figure 9.9 This shows the same graph as in Figure 9.6 except that 


we now suppose that the discrete variables z, are ob- 
served, as well as the data variables xn. m 


ha 
Now consider the problem of maximizing the likelihood for the complete data 
set {X, Z}. From (9.10) and (9.11), this likelihood function takes the form 


N K 
p(X, Ziu, D, m) = | | [| eet nlu Er) (9.35) 


n=1k=1 


where zng denotes the kt? component of zn. Taking the logarithm, we obtain 


N K 
In p(X, Z|u, ©, m) = > 5 Znk {ln Tmk + ln N (Xnl Hk, Ek) } (9.36) 


n=1 k=1 


Comparison with the log likelihood function (9.14) for the incomplete data shows 
that the summation over k and the logarithm have been interchanged. The loga- 
rithm now acts directly on the Gaussian distribution, which itself is a member of 
the exponential family. Not surprisingly, this leads to a much simpler solution to 
the maximum likelihood problem, as we now show. Consider first the maximization 
with respect to the means and covariances. Because z,, is a K-dimensional vec- 
tor with all elements equal to 0 except for a single element having the value 1, the 
complete-data log likelihood function is simply a sum of K independent contribu- 
tions, one for each mixture component. Thus the maximization with respect to a 
mean or a covariance is exactly as for a single Gaussian, except that it involves only 
the subset of data points that are ‘assigned’ to that component. For the maximization 
with respect to the mixing coefficients, we note that these are coupled for different 
values of k by virtue of the summation constraint (9.9). Again, this can be enforced 
using a Lagrange multiplier as before, and leads to the result 


1 N 
Tk = 3 Znk (9.37) 


so that the mixing coefficients are equal to the fractions of data points assigned to 
the corresponding components. 

Thus we see that the complete-data log likelihood function can be maximized 
trivially in closed form. In practice, however, we do not have values for the latent 
variables so, as discussed earlier, we consider the expectation, with respect to the 
posterior distribution of the latent variables, of the complete-data log likelihood. 
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Using (9.10) and (9.11) together with Bayes’ theorem, we see that this posterior 
distribution takes the form 


N K 


p(Z|X, u, £, 7) “l I [mN (Xn| pp, Er)” . (9.38) 


and hence factorizes over n so that under the posterior distribution the {z,,} are 
independent. This is easily verified by inspection of the directed graph in Figure 9.6 
and making use of the d-separation criterion. The expected value of the indicator 
variable z„x under this posterior distribution is then given by 


D enr [mN (nlt Dr) 


Znk 


Ep [TN (Xn |My £;)] ~ 


Znj 


Talio k 
Saw (xn|uj © j) 


which is just the responsibility of component k for data point x,,. The expected value 
of the complete-data log likelihood function is therefore given by 


E[Znk] = 


= 55.) (9.39) 


Ez(In p(X, Z|, £, 1)] = X So Wene) {In te + MN (Xn | fe, Be)}- (9.40) 


n=1 k=1 


We can now proceed as follows. First we choose some initial values for the param- 
eters y”, D”? and m’, and use these to evaluate the responsibilities (the E step). 
We then keep the responsibilities fixed and maximize (9.40) with respect to Hyp, Yk 
and my (the M step). This leads to closed form solutions for w"°”, H"°™ and w™ew 
given by (9.17), (9.19), and (9.22) as before. This is precisely the EM algorithm for 
Gaussian mixtures as derived earlier. We shall gain more insight into the role of the 
expected complete-data log likelihood function when we give a proof of convergence 
of the EM algorithm in Section 9.4. 


9.3.2 Relation to k-means 


Comparison of the K-means algorithm with the EM algorithm for Gaussian 
mixtures shows that there is a close similarity. Whereas the -means algorithm 
performs a hard assignment of data points to clusters, in which each data point is 
associated uniquely with one cluster, the EM algorithm makes a soft assignment 
based on the posterior probabilities. In fact, we can derive the K-means algorithm 
as a particular limit of EM for Gaussian mixtures as follows. 

Consider a Gaussian mixture model in which the covariance matrices of the 
mixture components are given by eI, where e is a variance parameter that is shared 
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by all of the components, and I is the identity matrix, so that 


1 1 
plain Bs) e E OD 


We now consider the EM algorithm for a mixture of K Gaussians of this form in 
which we treat € as a fixed constant, instead of a parameter to be re-estimated. From 
(9.13) the posterior probabilities, or responsibilities, for a particular data point Xn, 
are given by 
2 
ee exp {a Xn — Mell?/2e (9.42) 
5 Tj €XP { —||Xn = bs, ||?/2€} 

If we consider the limit € — 0, we see that in the denominator the term for which 
\|Xn — 44;||? is smallest will go to zero most slowly, and hence the responsibilities 
(Znk) for the data point x,, all go to zero except for term j, for which the responsi- 
bility 7(z,,;) will go to unity. Note that this holds independently of the values of the 
Tk SO long as none of the my is zero. Thus, in this limit, we obtain a hard assignment 
of data points to clusters, just as in the AK-means algorithm, so that y(znk) > Tnk 
where rnk is defined by (9.2). Each data point is thereby assigned to the cluster 
having the closest mean. 

The EM re-estimation equation for the u}, given by (9.17), then reduces to the 
k-means result (9.4). Note that the re-estimation formula for the mixing coefficients 
(9.22) simply re-sets the value of 7;, to be equal to the fraction of data points assigned 
to cluster k, although these parameters no longer play an active role in the algorithm. 

Finally, in the limit e — 0 the expected complete-data log likelihood, given by 
(9.40), becomes 


K 


N 
1 
Eg|In p(X, Z|, ¥,m)] > -3 >, > rnn — p,||? + const. (9.43) 


Thus we see that in this limit, maximizing the expected complete-data log likelihood 
is equivalent to minimizing the distortion measure J for the k-means algorithm 
given by (9.1). 

Note that the k-means algorithm does not estimate the covariances of the clus- 
ters but only the cluster means. A hard-assignment version of the Gaussian mixture 
model with general covariance matrices, known as the elliptical K -means algorithm, 
has been considered by Sung and Poggio (1994). 


9.3.3 Mixtures of Bernoulli distributions 


So far in this chapter, we have focussed on distributions over continuous vari- 
ables described by mixtures of Gaussians. As a further example of mixture mod- 
elling, and to illustrate the EM algorithm in a different context, we now discuss mix- 
tures of discrete binary variables described by Bernoulli distributions. This model 
is also known as latent class analysis (Lazarsfeld and Henry, 1968; McLachlan and 
Peel, 2000). As well as being of practical importance in its own right, our discus- 
sion of Bernoulli mixtures will also lay the foundation for a consideration of hidden 
Markov models over discrete variables. 
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Consider a set of D binary variables x;, where 7 = 1,..., D, each of which is 
governed by a Bernoulli distribution with parameter u;, so that 


pxu) = Ta 1 = al (9.44) 


where x = (z1,..., £p)" and w = (m1,..., up)". We see that the individual 
variables x; are independent, given yz. The mean and covariance of this distribution 
are easily seen to be 


Eix] = u (9.45) 
cov[x] = diag{pi(1— mi)}- (9.46) 


Now let us consider a finite mixture of these distributions given by 
p(x|u, r -Yan x|ur) (9.47) 
where u = {H;,..., Hg}, T ={m,...,7K}, and 


p(x|u,) = = Ti agg), (9.48) 


The mean and covariance of this mixture distribution are given by 


K 
Ex] = So mete (9.49) 

K 
cov[x] = Some {Dn + mni} — E>XE[x]" (9.50) 

k=1 
where X = diag {uxi(1 — fei) }. Because the covariance matrix cov[x] is no 


longer diagonal, the mixture distribution can capture correlations between the vari- 
ables, unlike a single Bernoulli distribution. 

If we are given a data set X = {x,,...,x} then the log likelihood function 
for this model is given by 


In p(X|p, 7 Sh {dona Xn| Hg) . (9.51) 
n=1 


Again we see the appearance of the summation inside the logarithm, so that the 
maximum likelihood solution no longer has closed form. 

We now derive the EM algorithm for maximizing the likelihood function for 
the mixture of Bernoulli distributions. To do this, we first introduce an explicit latent 
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variable z associated with each instance of x. As in the case of the Gaussian mixture, 
z = (21,--.-,2K)* is a binary K-dimensional variable having a single component 
equal to 1, with all other components equal to 0. We can then write the conditional 
distribution of x, given the latent variable, as 


p(x|z, p) - [In x|up)™ (9.52) 


while the prior distribution for the latent variables is the same as for the mixture of 
Gaussians model, so that 


plz\r) = II ae (9.53) 


If we form the product of p(x|z, p) and p(z|7r) and then marginalize over z, then we 
recover (9.47). 

In order to derive the EM algorithm, we first write down the complete-data log 
likelihood function, which is given by 


N K 
In p(X, Z|, 7) = DD rik fun 
n=1 k=1 


D 
+ rD [£ni ln Hki + (1 — £ni) ln(1 — mo} (9.54) 


i=1 


where X = {xn } and Z = {Zn }. Next we take the expectation of the complete-data 
log likelihood with respect to the posterior distribution of the latent variables to give 


N K 
Ezin p(X, Z|, m)| = `> Y Wend fi Tk 


n=1 k=1 


i=l 


D 


where ¥(Znx) = E[Zn x] is the posterior probability, or responsibility, of component 
k given data point x,,. In the E step, these responsibilities are evaluated using Bayes’ 
theorem, which takes the form 


> Znk [Tkp Enl u) 


Znk 


SO [rpne 


Znj 


TkP(Xn| My) 


k , 
Sm p(%n| Hy) 
j=l 


V(Znk) = Elng] 


(9.56) 
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If we consider the sum over n in (9.55), we see that the responsibilities enter 
only through two terms, which can be written as 


N 
Nk = S Wen) (9.57) 
n=1 
1 N 
X, = Ng de 1(2n8) Xn (9.58) 
n=1 


where I; is the effective number of data points associated with component k. In the 
M step, we maximize the expected complete-data log likelihood with respect to the 
parameters p, and m. If we set the derivative of (9.55) with respect to u, equal to 
zero and rearrange the terms, we obtain 


Ur = Fp. (9.59) 


We see that this sets the mean of component k equal to a weighted mean of the 
data, with weighting coefficients given by the responsibilities that component k takes 
for data points. For the maximization with respect to mk, we need to introduce a 
Lagrange multiplier to enforce the constraint X), mẹ = 1. Following analogous 
steps to those used for the mixture of Gaussians, we then obtain 


N (9.60) 


Tk 
which represents the intuitively reasonable result that the mixing coefficient for com- 
ponent k is given by the effective fraction of points in the data set explained by that 
component. 

Note that in contrast to the mixture of Gaussians, there are no singularities in 
which the likelihood function goes to infinity. This can be seen by noting that the 
likelihood function is bounded above because 0 < p(Xn|uĶ) < 1. There exist 
singularities at which the likelihood function goes to zero, but these will not be 
found by EM provided it is not initialized to a pathological starting point, because 
the EM algorithm always increases the value of the likelihood function, until a local 
maximum is found. We illustrate the Bernoulli mixture model in Figure 9.10 by 
using it to model handwritten digits. Here the digit images have been turned into 
binary vectors by setting all elements whose values exceed 0.5 to 1 and setting the 
remaining elements to 0. We now fit a data set of N = 600 such digits, comprising 
the digits ‘2’, ‘3’, and ‘4’, with a mixture of K = 3 Bernoulli distributions by 
running 10 iterations of the EM algorithm. The mixing coefficients were initialized 
to Tk = 1/K, and the parameters uy; were set to random values chosen uniformly in 
the range (0.25, 0.75) and then normalized to satisfy the constraint that Be Ukj = 1. 
We see that a mixture of 3 Bernoulli distributions is able to find the three clusters in 
the data set corresponding to the different digits. 

The conjugate prior for the parameters of a Bernoulli distribution is given by 
the beta distribution, and we have seen that a beta prior is equivalent to introducing 
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Figure 9.10 Illustration of the Bernoulli mixture model in which the top row shows examples from the digits data 
set after converting the pixel values from grey scale to binary using a threshold of 0.5. On the bottom row the first 
three images show the parameters jx; for each of the three components in the mixture model. As a comparison, 
we also fit the same data set using a single multivariate Bernoulli distribution, again using maximum likelihood. 
This amounts to simply averaging the counts in each pixel and is shown by the right-most image on the bottom 


row. 
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additional effective observations of x. We can similarly introduce priors into the 
Bernoulli mixture model, and use EM to maximize the posterior probability distri- 
butions. 

It is straightforward to extend the analysis of Bernoulli mixtures to the case of 
multinomial binary variables having M > 2 states by making use of the discrete dis- 
tribution (2.26). Again, we can introduce Dirichlet priors over the model parameters 
if desired. 


9.3.4 EM for Bayesian linear regression 


As a third example of the application of EM, we return to the evidence ap- 
proximation for Bayesian linear regression. In Section 3.5.2, we obtained the re- 
estimation equations for the hyperparameters a and (3 by evaluation of the evidence 
and then setting the derivatives of the resulting expression to zero. We now turn to 
an alternative approach for finding a and 8 based on the EM algorithm. Recall that 
our goal is to maximize the evidence function p(t|a, 8) given by (3.77) with respect 
to a and 8. Because the parameter vector w is marginalized out, we can regard it as 
a latent variable, and hence we can optimize this marginal likelihood function using 
EM. In the E step, we compute the posterior distribution of w given the current set- 
ting of the parameters a and 8 and then use this to find the expected complete-data 
log likelihood. In the M step, we maximize this quantity with respect to a and 8. We 
have already derived the posterior distribution of w because this is given by (3.49). 
The complete-data log likelihood function is then given by 


In p(t, wla, 3) = ln p(t|w, 8) + In p(wia) (9.61) 
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where the likelihood p(t|w, 3) and the prior p(w|q) are given by (3.10) and (3.52), 
respectively, and y(x, w) is given by (3.3). Taking the expectation with respect to 
the posterior distribution of w then gives 


M N 
PEE Ee), = n (5) 2E a (2) 
N 
2 DOE [(tn —w"d,,)’] - (9.62) 
n=1 


Setting the derivatives with respect to a to zero, we obtain the M step re-estimation 
equation 
M M 


E [wT w] = mymy + Tr(Sw) 


(9.63) 


a= 


An analogous result holds for £. 

Note that this re-estimation equation takes a slightly different form from the 
corresponding result (3.92) derived by direct evaluation of the evidence function. 
However, they each involve computation and inversion (or eigen decomposition) of 
an M x M matrix and hence will have comparable computational cost per iteration. 

These two approaches to determining a should of course converge to the same 
result (assuming they find the same local maximum of the evidence function). This 
can be verified by first noting that the quantity y is defined by 


M 
y=M-a)y_ Yara =M- TSn). (9.64) 
i=l 


At a stationary point of the evidence function, the re-estimation equation (3.92) will 
be self-consistently satisfied, and hence we can substitute for y to give 


amymy = 7 = M —aTr(Sy) (9.65) 


and solving for œ we obtain (9.63), which is precisely the EM re-estimation equation. 

As a final example, we consider a closely related model, namely the relevance 
vector machine for regression discussed in Section 7.2.1. There we used direct max- 
imization of the marginal likelihood to derive re-estimation equations for the hyper- 
parameters a and 8. Here we consider an alternative approach in which we view the 
weight vector w as a latent variable and apply the EM algorithm. The E step involves 
finding the posterior distribution over the weights, and this is given by (7.81). In the 
M step we maximize the expected complete-data log likelihood, which is defined by 


Ew (In p(t|X, w, 3)p(w|a)| (9.66) 


where the expectation is taken with respect to the posterior distribution computed 
using the ‘old’ parameter values. To compute the new parameter values we maximize 
with respect to œ and £ to give 
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new _ 1 
new)—1 It- Omy |)? +B" Viv 
(P) x 


(9.68) 


These re-estimation equations are formally equivalent to those obtained by direct 
maxmization. 


The EM Algorithm in General 


The expectation maximization algorithm, or EM algorithm, is a general technique for 
finding maximum likelihood solutions for probabilistic models having latent vari- 
ables (Dempster et al., 1977; McLachlan and Krishnan, 1997). Here we give a very 
general treatment of the EM algorithm and in the process provide a proof that the 
EM algorithm derived heuristically in Sections 9.2 and 9.3 for Gaussian mixtures 
does indeed maximize the likelihood function (Csiszar and Tusnady, 1984; Hath- 
away, 1986; Neal and Hinton, 1999). Our discussion will also form the basis for the 
derivation of the variational inference framework. 

Consider a probabilistic model in which we collectively denote all of the ob- 
served variables by X and all of the hidden variables by Z. The joint distribution 
p(X, Z|@) is governed by a set of parameters denoted 0. Our goal is to maximize 
the likelihood function that is given by 


p(X|0) = X` K(X, Z|). (9.69) 


Here we are assuming Z is discrete, although the discussion is identical if Z com- 
prises continuous variables or a combination of discrete and continuous variables, 
with summation replaced by integration as appropriate. 

We shall suppose that direct optimization of p(X|0) is difficult, but that opti- 
mization of the complete-data likelihood function p(X, Z|@) is significantly easier. 
Next we introduce a distribution q(Z) defined over the latent variables, and we ob- 
serve that, for any choice of q(Z), the following decomposition holds 


In p(X|@) = L(g, A) + KL(q||p) (9.70) 

where we have defined 
L(q,0) = Satan {PAO | (9.71) 
KL(qllp) = -` q(Z)In eae} l (9.72) 


Note that £(q,@) is a functional (see Appendix D for a discussion of functionals) 
of the distribution q(Z), and a function of the parameters 0. It is worth studying 
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Illustration of the decomposition given 
by (9.70), which holds for any choice 
of distribution q(Z). Because the KL 
Kullback-Leibler divergence satisfies (allp) 
KL(q||p) > 0, we see that the quan- 
tity L (q, 8) is a lower bound on the log 
likelihood function In p(X|98). 


carefully the forms of the expressions (9.71) and (9.72), and in particular noting that 
they differ in sign and also that £(q, 0) contains the joint distribution of X and Z 
while KL(q||p) contains the conditional distribution of Z given X. To verify the 
decomposition (9.70), we first make use of the product rule of probability to give 


In p(X, Z|@) = In p(Z|X, 0) + In p(X]0) (9.73) 


which we then substitute into the expression for £(q, 0). This gives rise to two terms, 
one of which cancels KL(q||p) while the other gives the required log likelihood 
In p(X|@) after noting that q(Z) is a normalized distribution that sums to 1. 

From (9.72), we see that KL(q||p) is the Kullback-Leibler divergence between 
q(Z) and the posterior distribution p(Z|X, @). Recall that the Kullback-Leibler di- 
vergence satisfies KL(q||p) > 0, with equality if, and only if, g(Z) = p(Z|X, 0). It 
therefore follows from (9.70) that £(q, 0) < In p(X), in other words that L(q, 0) 
is a lower bound on Inp(X|@). The decomposition (9.70) is illustrated in Fig- 
ure 9.11. 

The EM algorithm is a two-stage iterative optimization technique for finding 
maximum likelihood solutions. We can use the decomposition (9.70) to define the 
EM algorithm and to demonstrate that it does indeed maximize the log likelihood. 
Suppose that the current value of the parameter vector is @°'". In the E step, the 
lower bound L(q, 0°“) is maximized with respect to q(Z) while holding 6°" fixed. 
The solution to this maximization problem is easily seen by noting that the value 
of In p(X|@°") does not depend on q(Z) and so the largest value of £(q, 0°") will 
occur when the Kullback-Leibler divergence vanishes, in other words when q(Z) is 
equal to the posterior distribution p(Z|X, 0°"). In this case, the lower bound will 
equal the log likelihood, as illustrated in Figure 9.12. 

In the subsequent M step, the distribution q(Z) is held fixed and the lower bound 
L(q,@) is maximized with respect to @ to give some new value 6"°”. This will 
cause the lower bound £ to increase (unless it is already at a maximum), which will 
necessarily cause the corresponding log likelihood function to increase. Because the 
distribution g is determined using the old parameter values rather than the new values 
and is held fixed during the M step, it will not equal the new posterior distribution 
p(Z|X, 0™°™), and hence there will be a nonzero KL divergence. The increase in the 
log likelihood function is therefore greater than the increase in the lower bound, as 
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Figure 9.12 


Figure 9.13 


Illustration of the E step of KL(q||p) =0 
the EM algorithm. The q 
distribution is set equal to 
the posterior distribution for 
the current parameter val- 
ues 6%, causing the lower 
bound to move up to the 
same value as the log like- 
lihood function, with the KL 


divergence vanishing. In p(X|0°"") 


shown in Figure 9.13. If we substitute q(Z) = p(Z|X, 0°") into (9.71), we see that, 
after the E step, the lower bound takes the form 


£(q,0) = SY -p(Z|X,0") np(X, ZA) — X` p(Z|X, 6°") In p(Z|X, 6°") 
Z Z 
= Q(0,0°") + const (9.74) 


where the constant is simply the negative entropy of the q distribution and is there- 
fore independent of 0. Thus in the M step, the quantity that is being maximized is the 
expectation of the complete-data log likelihood, as we saw earlier in the case of mix- 
tures of Gaussians. Note that the variable 0 over which we are optimizing appears 
only inside the logarithm. If the joint distribution p(Z, X|@) comprises a member of 
the exponential family, or a product of such members, then we see that the logarithm 
will cancel the exponential and lead to an M step that will be typically much simpler 
than the maximization of the corresponding incomplete-data log likelihood function 
p(X]0). 

The operation of the EM algorithm can also be viewed in the space of parame- 
ters, as illustrated schematically in Figure 9.14. Here the red curve depicts the (in- 


Illustration of the M step of the EM 
algorithm. The distribution (Z) KL(q||p) 
is held fixed and the lower bound 
L(q,9) is maximized with respect 
to the parameter vector @ to give 
a revised value 8”°”. Because the 
KL divergence is nonnegative, this 
causes the log likelihood In p(X|@) 
to increase by at least as much as 
the lower bound does. 


L(q,0"") In p(X|a"™) 


Figure 9.14 The EM algorithm involves alter- 
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nately computing a lower bound 
on the log likelihood for the cur- 
rent parameter values and then 
maximizing this bound to obtain 
the new parameter values. See 
the text for a full discussion. 


gold grew 


complete data) log likelihood function whose value we wish to maximize. We start 
with some initial parameter value 0°", and in the first E step we evaluate the poste- 
rior distribution over latent variables, which gives rise to a lower bound £(0, 009) 
whose value equals the log likelihood at gl) as shown by the blue curve. Note that 
the bound makes a tangential contact with the log likelihood at 0, so that both 
curves have the same gradient. This bound is a convex function having a unique 
maximum (for mixture components from the exponential family). In the M step, the 
bound is maximized giving the value 9, which gives a larger value of log likeli- 
hood than 0°!) The subsequent E step then constructs a bound that is tangential at 
ge) as shown by the green curve. 

For the particular case of an independent, identically distributed data set, X 
will comprise N data points {xn} while Z will comprise N corresponding latent 
variables {z,,}, where n = 1,..., N. From the independence assumption, we have 
p(X, Z) = [],, p(Xn,Zn) and, by marginalizing over the {z,,} we have p(X) = 
[I,, p(Xn). Using the sum and product rules, we see that the posterior probability 
that is evaluated in the E step takes the form 


N 
a [| 2&n. 2n!9) N 
p(X, Z|@) Zam = [[ plenlxn,@) (9.75) 


p(X, ZIO) 
> 


p(Z|X, 4) _ 


n=1 


P(Xn; Zn|9) 


n=1 


and so the posterior distribution also factorizes with respect to n. In the case of 
the Gaussian mixture model this simply says that the responsibility that each of the 
mixture components takes for a particular data point x,, depends only on the value 
of Xn and on the parameters 0 of the mixture components, not on the values of the 
other data points. 

We have seen that both the E and the M steps of the EM algorithm are increas- 
ing the value of a well-defined bound on the log likelihood function and that the 
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complete EM cycle will change the model parameters in such a way as to cause 
the log likelihood to increase (unless it is already at a maximum, in which case the 
parameters remain unchanged). 

We can also use the EM algorithm to maximize the posterior distribution p(6|X) 
for models in which we have introduced a prior p(@) over the parameters. To see this, 
we note that as a function of 0, we have p(@|X) = p(@, X)/p(X) and so 


In p(@|X) = In p(@, X) — In p(X). (9.76) 
Making use of the decomposition (9.70), we have 


Inp(@|X) = L(g,0) + KL(q||p) + np(@) — In p(X) 
> L(q,0) + Inp(@) — Inp(X). (9.77) 


where In p(X) is a constant. We can again optimize the right-hand side alternately 
with respect to q and 0. The optimization with respect to q gives rise to the same E- 
step equations as for the standard EM algorithm, because q only appears in £(q, 0). 
The M-step equations are modified through the introduction of the prior term In p(8), 
which typically requires only a small modification to the standard maximum likeli- 
hood M-step equations. 

The EM algorithm breaks down the potentially difficult problem of maximizing 
the likelihood function into two stages, the E step and the M step, each of which will 
often prove simpler to implement. Nevertheless, for complex models it may be the 
case that either the E step or the M step, or indeed both, remain intractable. This 
leads to two possible extensions of the EM algorithm, as follows. 

The generalized EM, or GEM, algorithm addresses the problem of an intractable 
M step. Instead of aiming to maximize £(q,@) with respect to 0, it seeks instead 
to change the parameters in such a way as to increase its value. Again, because 
L(q, 0) is a lower bound on the log likelihood function, each complete EM cycle of 
the GEM algorithm is guaranteed to increase the value of the log likelihood (unless 
the parameters already correspond to a local maximum). One way to exploit the 
GEM approach would be to use one of the nonlinear optimization strategies, such 
as the conjugate gradients algorithm, during the M step. Another form of GEM 
algorithm, known as the expectation conditional maximization, or ECM, algorithm, 
involves making several constrained optimizations within each M step (Meng and 
Rubin, 1993). For instance, the parameters might be partitioned into groups, and the 
M step is broken down into multiple steps each of which involves optimizing one of 
the subset with the remainder held fixed. 

We can similarly generalize the E step of the EM algorithm by performing a 
partial, rather than complete, optimization of £(q, 0) with respect to q(Z) (Neal and 
Hinton, 1999). As we have seen, for any given value of @ there is a unique maximum 
of L(q, 0) with respect to q(Z) that corresponds to the posterior distribution gg(Z) = 
p(Z|X,@) and that for this choice of q(Z) the bound £(q,0) is equal to the log 
likelihood function In p(X|@). It follows that any algorithm that converges to the 
global maximum of £(q, 0) will find a value of @ that is also a global maximum 
of the log likelihood In p(X|@). Provided p(X, Z|@) is a continuous function of 0 
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then, by continuity, any local maximum of £(q, 0) will also be a local maximum of 


In p(X|6). 
Consider the case of N independent data points x,,...,xj with corresponding 
latent variables z1,...,z. The joint distribution p(X, Z|@) factorizes over the data 


points, and this structure can be exploited in an incremental form of EM in which 
at each EM cycle only one data point is processed at a time. In the E step, instead 
of recomputing the responsibilities for all of the data points, we just re-evaluate the 
responsibilities for one data point. It might appear that the subsequent M step would 
require computation involving the responsibilities for all of the data points. How- 
ever, if the mixture components are members of the exponential family, then the 
responsibilities enter only through simple sufficient statistics, and these can be up- 
dated efficiently. Consider, for instance, the case of a Gaussian mixture, and suppose 
we perform an update for data point m in which the corresponding old and new 
values of the responsibilities are denoted y°!4(z),,) and y"°" (zm). In the M step, 
the required sufficient statistics can be updated incrementally. For instance, for the 
means the sufficient statistics are defined by (9.17) and (9.18) from which we obtain 


new old 
new O y K k = y K k O 
u? = pee ( ( m ( m ) (Xm = pe) (9.78) 
k 
together with 
Ne 2 Np i YPY (aa) = Y zmk). (9.79) 
The corresponding results for the covariances and the mixing coefficients are analo- 


gous. 

Thus both the E step and the M step take fixed time that is independent of the 
total number of data points. Because the parameters are revised after each data point, 
rather than waiting until after the whole data set is processed, this incremental ver- 
sion can converge faster than the batch version. Each E or M step in this incremental 
algorithm is increasing the value of £(q,@) and, as we have shown above, if the 
algorithm converges to a local (or global) maximum of £(q, 0), this will correspond 
to a local (or global) maximum of the log likelihood function In p(X|@). 


(x) FQ Consider the k-means algorithm discussed in Section 9.1. Show that as 
a consequence of there being a finite number of possible assignments for the set of 
discrete indicator variables r», and that for each such assignment there is a unique 
optimum for the { u, }, the K-means algorithm must converge after a finite number 
of iterations. 


(x) Apply the Robbins-Monro sequential estimation procedure described in Sec- 
tion 2.3.5 to the problem of finding the roots of the regression function given by 
the derivatives of J in (9.1) with respect to wp. Show that this leads to a stochastic 
K-means algorithm in which, for each data point xp, the nearest prototype ju, is 
updated using (9.5). 
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9.3 


9.4 


9.5 


9.6 


9.7 


9.8 


9.9 


9.10 


(x) EA Consider a Gaussian mixture model in which the marginal distribution 
p(z) for the latent variable is given by (9.10), and the conditional distribution p(x|z) 
for the observed variable is given by (9.11). Show that the marginal distribution 
p(x), obtained by summing p(z)p(x|z) over all possible values of z, is a Gaussian 
mixture of the form (9.7). 


(x) Suppose we wish to use the EM algorithm to maximize the posterior distri- 
bution over parameters p(0|X) for a model containing latent variables, where X is 
the observed data set. Show that the E step remains the same as in the maximum 
likelihood case, whereas in the M step the quantity to be maximized is given by 
Q(0, 0%?) + In p(@) where Q(0, 0°") is defined by (9.30). 


(x) Consider the directed graph for a Gaussian mixture model shown in Figure 9.6. 
By making use of the d-separation criterion discussed in Section 8.2, show that the 
posterior distribution of the latent variables factorizes with respect to the different 
data points so that 


N 
p(Z|X, u, £, 7) = EEN E, r). (9.80) 


n=l 


(xx) Consider a special case of a Gaussian mixture model in which the covari- 
ance matrices Xy of the components are all constrained to have a common value 
X. Derive the EM equations for maximizing the likelihood function under such a 
model. 


(x) EU Verify that maximization of the complete-data log likelihood (9.36) for 
a Gaussian mixture model leads to the result that the means and covariances of each 
component are fitted independently to the corresponding group of data points, and 
the mixing coefficients are given by the fractions of points in each group. 


(x) FY Show that if we maximize (9.40) with respect to p, while keeping the 
responsibilities y(zn,) fixed, we obtain the closed form solution given by (9.17). 


(x) Show that if we maximize (9.40) with respect to Xy and 7; while keeping the 
responsibilities y(z,%) fixed, we obtain the closed form solutions given by (9.19) 
and (9.22). 


(xx) Consider a density model given by a mixture distribution 


K 
p(x) = X mpak) (9.81) 
k=1 
and suppose that we partition the vector x into two parts so that x = (Xa, X»). 


Show that the conditional density p(x,|xq) is itself a mixture distribution and find 
expressions for the mixing coefficients and for the component densities. 


9.11 


9.12 


9.13 


9.14 


9.15 


9.16 


9.17 
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(x) In Section 9.3.2, we obtained a relationship between K means and EM for 
Gaussian mixtures by considering a mixture model in which all components have 
covariance eI. Show that in the limit €e — 0, maximizing the expected complete- 
data log likelihood for this model, given by (9.40), is equivalent to minimizing the 
distortion measure J for the k-means algorithm given by (9.1). 


œ FEY Consider a mixture distribution of the form 


K 
p(x) = 5 > mep(x|k) (9.82) 
k=1 


where the elements of x could be discrete or continuous or a combination of these. 
Denote the mean and covariance of p(x|k) by u, and Xg, respectively. Show that 
the mean and covariance of the mixture distribution are given by (9.49) and (9.50). 


(xx) Using the re-estimation equations for the EM algorithm, show that a mix- 
ture of Bernoulli distributions, with its parameters set to values corresponding to a 
maximum of the likelihood function, has the property that 


1 N 


Bix] = — X xn =R. (9.83) 


n=1 


Hence show that if the parameters of this model are initialized such that all compo- 
nents have the same mean u, = f for k = 1,..., K, then the EM algorithm will 
converge after one iteration, for any choice of the initial mixing coefficients, and that 
this solution has the property jz; = X. Note that this represents a degenerate case of 
the mixture model in which all of the components are identical, and in practice we 
try to avoid such solutions by using an appropriate initialization. 


(x) Consider the joint distribution of latent and observed variables for the Bernoulli 
distribution obtained by forming the product of p(x|z, p) given by (9.52) and p(z|7) 
given by (9.53). Show that if we marginalize this joint distribution with respect to z, 
then we obtain (9.47). 


œ FY Show that if we maximize the expected complete-data log likelihood 
function (9.55) for a mixture of Bernoulli distributions with respect to uy, we obtain 
the M step equation (9.59). 


(x) Show that if we maximize the expected complete-data log likelihood function 
(9.55) for a mixture of Bernoulli distributions with respect to the mixing coefficients 
Tk, using a Lagrange multiplier to enforce the summation constraint, we obtain the 
M step equation (9.60). 


(x) FY Show that as a consequence of the constraint 0 < p(xp|M,) < 1 for 
the discrete variable x,,, the incomplete-data log likelihood function for a mixture 
of Bernoulli distributions is bounded above, and hence that there are no singularities 
for which the likelihood goes to infinity. 
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9.18 


9.19 


9.20 


9.21 


9.22 


9.23 


9.24 


(xx) Consider a Bernoulli mixture model as discussed in Section 9.3.3, together 
with a prior distribution p(j1;,,|ax, bk) over each of the parameter vectors py, given 
by the beta distribution (2.13), and a Dirichlet prior p(|a) given by (2.38). Derive 
the EM algorithm for maximizing the posterior probability p(s, 7|X). 


(xx) Consider a D-dimensional variable x each of whose components 2 is itself a 
multinomial variable of degree M so that x is a binary vector with components £i; 
where? = 1,..., D and j = 1,..., M, subject to the constraint that DoF, Zij = 1 for 
all 7. Suppose that the distribution of these variables is described by a mixture of the 
discrete multinomial distributions considered in Section 2.2 so that 


K 
p(x) = X rep (|e) (9.84) 
k=1 
where 
DM 
rel= end. (9.85) 
i=1 j=l 
The parameters ji%;; represent the probabilities p(x;; = 1|up) and must satisfy 
0 < Ukij < 1 together with the constraint ar Ukij = 1 for all values of k and i. 
Given an observed data set {x,,}, where n = 1,...,.N, derive the E and M step 


equations of the EM algorithm for optimizing the mixing coefficients mẹ and the 
component parameters j;,;; of this distribution by maximum likelihood. 


(x) [QTY Show that maximization of the expected complete-data log likelihood 
function (9.62) for the Bayesian linear regression model leads to the M step re- 
estimation result (9.63) for a. 


(xx) Using the evidence framework of Section 3.5, derive the M-step re-estimation 
equations for the parameter ( in the Bayesian linear regression model, analogous to 
the result (9.63) for a. 


(xx) By maximization of the expected complete-data log likelihood defined by 
(9.66), derive the M step equations (9.67) and (9.68) for re-estimating the hyperpa- 
rameters of the relevance vector machine for regression. 


(« x) | www | In Section 7.2.1 we used direct maximization of the marginal like- 
lihood to derive the re-estimation equations (7.87) and (7.88) for finding values of 
the hyperparameters œ and 8 for the regression RVM. Similarly, in Section 9.3.4 
we used the EM algorithm to maximize the same marginal likelihood, giving the 
re-estimation equations (9.67) and (9.68). Show that these two sets of re-estimation 
equations are formally equivalent. 


(x) Verify the relation (9.70) in which £(q, 0) and KL(gq||p) are defined by (9.71) 
and (9.72), respectively. 
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9.25 (x) [QQ Show that the lower bound L(q,4) given by (9.71), with g(Z) = 
p(Z|X, 0d), has the same gradient with respect to 0 as the log likelihood function 
In p(X|@) at the point 0 = a), 


9.26 (x) PQQ Consider the incremental form of the EM algorithm for a mixture of 
Gaussians, in which the responsibilities are recomputed only for a specific data point 
Xm. Starting from the M-step formulae (9.17) and (9.18), derive the results (9.78) 
and (9.79) for updating the component means. 


9.27 (xx) Derive M-step formulae for updating the covariance matrices and mixing 
coefficients in a Gaussian mixture model when the responsibilities are updated in- 
crementally, analogous to the result (9.78) for updating the means. 


10 


Approximate 


“4 


A p> _* 


A central task in the application of probabilistic models is the evaluation of the pos- 
terior distribution p(Z|X) of the latent variables Z given the observed (visible) data 
variables X, and the evaluation of expectations computed with respect to this dis- 
tribution. The model might also contain some deterministic parameters, which we 
will leave implicit for the moment, or it may be a fully Bayesian model in which any 
unknown parameters are given prior distributions and are absorbed into the set of 
latent variables denoted by the vector Z. For instance, in the EM algorithm we need 
to evaluate the expectation of the complete-data log likelihood with respect to the 
posterior distribution of the latent variables. For many models of practical interest, it 
will be infeasible to evaluate the posterior distribution or indeed to compute expec- 
tations with respect to this distribution. This could be because the dimensionality of 
the latent space is too high to work with directly or because the posterior distribution 
has a highly complex form for which expectations are not analytically tractable. In 
the case of continuous variables, the required integrations may not have closed-form 
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10.1. 


analytical solutions, while the dimensionality of the space and the complexity of the 
integrand may prohibit numerical integration. For discrete variables, the marginal- 
izations involve summing over all possible configurations of the hidden variables, 
and though this is always possible in principle, we often find in practice that there 
may be exponentially many hidden states so that exact calculation is prohibitively 
expensive. 

In such situations, we need to resort to approximation schemes, and these fall 
broadly into two classes, according to whether they rely on stochastic or determin- 
istic approximations. Stochastic techniques such as Markov chain Monte Carlo, de- 
scribed in Chapter 11, have enabled the widespread use of Bayesian methods across 
many domains. They generally have the property that given infinite computational 
resource, they can generate exact results, and the approximation arises from the use 
of a finite amount of processor time. In practice, sampling methods can be compu- 
tationally demanding, often limiting their use to small-scale problems. Also, it can 
be difficult to know whether a sampling scheme is generating independent samples 
from the required distribution. 

In this chapter, we introduce a range of deterministic approximation schemes, 
some of which scale well to large applications. These are based on analytical ap- 
proximations to the posterior distribution, for example by assuming that it factorizes 
in a particular way or that it has a specific parametric form such as a Gaussian. As 
such, they can never generate exact results, and so their strengths and weaknesses 
are complementary to those of sampling methods. 

In Section 4.4, we discussed the Laplace approximation, which is based on a 
local Gaussian approximation to a mode (i.e., a maximum) of the distribution. Here 
we turn to a family of approximation techniques called variational inference or vari- 
ational Bayes, which use more global criteria and which have been widely applied. 
We conclude with a brief introduction to an alternative variational framework known 
as expectation propagation. 


Variational Inference 


Variational methods have their origins in the 18‘" century with the work of Euler, 
Lagrange, and others on the calculus of variations. Standard calculus is concerned 
with finding derivatives of functions. We can think of a function as a mapping that 
takes the value of a variable as the input and returns the value of the function as the 
output. The derivative of the function then describes how the output value varies 
as we make infinitesimal changes to the input value. Similarly, we can define a 
functional as a mapping that takes a function as the input and that returns the value 
of the functional as the output. An example would be the entropy H[p], which takes 
a probability distribution p(x) as the input and returns the quantity 


a= f PEET (10.1) 
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as the output. We can the introduce the concept of a functional derivative, which ex- 
presses how the value of the functional changes in response to infinitesimal changes 
to the input function (Feynman et al., 1964). The rules for the calculus of variations 
mirror those of standard calculus and are discussed in Appendix D. Many problems 
can be expressed in terms of an optimization problem in which the quantity being 
optimized is a functional. The solution is obtained by exploring all possible input 
functions to find the one that maximizes, or minimizes, the functional. Variational 
methods have broad applicability and include such areas as finite element methods 
(Kapur, 1989) and maximum entropy (Schwarz, 1988). 

Although there is nothing intrinsically approximate about variational methods, 
they do naturally lend themselves to finding approximate solutions. This is done 
by restricting the range of functions over which the optimization is performed, for 
instance by considering only quadratic functions or by considering functions com- 
posed of a linear combination of fixed basis functions in which only the coefficients 
of the linear combination can vary. In the case of applications to probabilistic in- 
ference, the restriction may for example take the form of factorization assumptions 
(Jordan et al., 1999; Jaakkola, 2001). 

Now let us consider in more detail how the concept of variational optimization 
can be applied to the inference problem. Suppose we have a fully Bayesian model in 
which all parameters are given prior distributions. The model may also have latent 
variables as well as parameters, and we shall denote the set of all latent variables 
and parameters by Z. Similarly, we denote the set of all observed variables by X. 
For example, we might have a set of N independent, identically distributed data, 
for which X = {x1,...,xw} and Z = {z1,...,zn}. Our probabilistic model 
specifies the joint distribution p(X, Z), and our goal is to find an approximation for 
the posterior distribution p(Z|X) as well as for the model evidence p(X). As in our 
discussion of EM, we can decompose the log marginal probability using 


In p(X) = £(q) + KL(q||p) (10.2) 


where we have defined 


L(q) = [a dZ (10.3) 


- [amn [EA dZ. (10.4) 


This differs from our discussion of EM only in that the parameter vector 0 no longer 
appears, because the parameters are now stochastic variables and are absorbed into 
Z. Since in this chapter we will mainly be interested in continuous variables we have 
used integrations rather than summations in formulating this decomposition. How- 
ever, the analysis goes through unchanged if some or all of the variables are discrete 
simply by replacing the integrations with summations as required. As before, we 
can maximize the lower bound £(q) by optimization with respect to the distribution 
q(Z), which is equivalent to minimizing the KL divergence. If we allow any possible 
choice for q(Z), then the maximum of the lower bound occurs when the KL diver- 
gence vanishes, which occurs when q(Z) equals the posterior distribution p(Z|X). 


KL(q||p) 
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Illustration of the variational approximation for the example considered earlier in Figure 4.14. The 


left-hand plot shows the original distribution (yellow) along with the Laplace (red) and variational (green) approx- 
imations, and the right-hand plot shows the negative logarithms of the corresponding curves. 


However, we shall suppose the model is such that working with the true posterior 
distribution is intractable. 

We therefore consider instead a restricted family of distributions g(Z) and then 
seek the member of this family for which the KL divergence is minimized. Our goal 
is to restrict the family sufficiently that they comprise only tractable distributions, 
while at the same time allowing the family to be sufficiently rich and flexible that it 
can provide a good approximation to the true posterior distribution. It is important to 
emphasize that the restriction is imposed purely to achieve tractability, and that sub- 
ject to this requirement we should use as rich a family of approximating distributions 
as possible. In particular, there is no ‘over-fitting’ associated with highly flexible dis- 
tributions. Using more flexible approximations simply allows us to approach the true 
posterior distribution more closely. 

One way to restrict the family of approximating distributions is to use a paramet- 
ric distribution q(Z|w) governed by a set of parameters w. The lower bound L(q) 
then becomes a function of w, and we can exploit standard nonlinear optimization 
techniques to determine the optimal values for the parameters. An example of this 
approach, in which the variational distribution is a Gaussian and we have optimized 
with respect to its mean and variance, is shown in Figure 10.1. 


10.1.1 Factorized distributions 


Here we consider an alternative way in which to restrict the family of distri- 
butions q(Z). Suppose we partition the elements of Z into disjoint groups that we 
denote by Z; where i = 1,..., M. We then assume that the q distribution factorizes 
with respect to these groups, so that 


(Z) = | | a:(Z:). (10.5) 


10.1. Variational Inference 465 


It should be emphasized that we are making no further assumptions about the distri- 
bution. In particular, we place no restriction on the functional forms of the individual 
factors q;(Z;). This factorized form of variational inference corresponds to an ap- 
proximation framework developed in physics called mean field theory (Parisi, 1988). 

Amongst all distributions q(Z) having the form (10.5), we now seek that distri- 
bution for which the lower bound L(q) is largest. We therefore wish to make a free 
form (variational) optimization of £ (q) with respect to all of the distributions q;(Z;), 
which we do by optimizing with respect to each of the factors in turn. To achieve 
this, we first substitute (10.5) into (10.3) and then dissect out the dependence on one 
of the factors q;(Z;). Denoting q;(Z;) by simply q; to keep the notation uncluttered, 
we then obtain 


L(q) 


Jie {a2 (X,Z) -Zuel dZ 


fo { fms z aaz} dZ; — fuma dZ; + const 


iA 


where we have defined a new distribution p(X, Z;) by the relation 


Inp(X, Z;) = Rij (In p(X, Z)| + const. (10.7) 


Here the notation E;~;|- -- ] denotes an expectation with respect to the q distributions 
over all variables z; for i Æ 7, so that 


E;z;[In p(X, Z)] = / Inp(X,Z) | [| aaz.. (10.8) 
ifj 


Now suppose we keep the {q;z;} fixed and maximize £(q) in (10.6) with re- 
spect to all possible forms for the distribution q;(Z;). This is easily done by rec- 
ognizing that (10.6) is a negative Kullback-Leibler divergence between q;(Z;) and 
p(X, Z;). Thus maximizing (10.6) is equivalent to minimizing the Kullback-Leibler 


Leonhard Euler contributions, he formulated the modern theory of the 
1707-1783 function, he developed (together with Lagrange) the 

calculus of variations, and he discovered the formula 
Euler was a Swiss mathematician e¢’” = —1, which relates four of the most important 


and physicist who worked in St. numbers in mathematics. During the last 17 years of 
Petersburg and Berlin and who is his life, he was almost totally blind, and yet he pro- 
widely considered to be one of the duced nearly half of his results during this period. 
greatest mathematicians of all time. 

He is certainly the most prolific, and 

his collected works fill 75 volumes. Amongst his many 
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divergence, and the minimum occurs when q;(Z;) = p(X, Z,;). Thus we obtain a 
general expression for the optimal solution q7(Z;) given by 


Ing (Z;) = Eiz;[In p(X, Z)] + const. (10.9) 


It is worth taking a few moments to study the form of this solution as it provides the 
basis for applications of variational methods. It says that the log of the optimal so- 
lution for factor qj is obtained simply by considering the log of the joint distribution 
over all hidden and visible variables and then taking the expectation with respect to 
all of the other factors {q;} for i Æ j. 

The additive constant in (10.9) is set by normalizing the distribution qj (Zj). 
Thus if we take the exponential of both sides and normalize, we have 


oes talp) 
[ox (E:z;[In p(X, Z)]) dZ; 


— 


exp 


In practice, we shall find it more convenient to work with the form (10.9) and then re- 
instate the normalization constant (where required) by inspection. This will become 
clear from subsequent examples. 

The set of equations given by (10.9) for 7 = 1,..., M represent a set of con- 
sistency conditions for the maximum of the lower bound subject to the factorization 
constraint. However, they do not represent an explicit solution because the expres- 
sion on the right-hand side of (10.9) for the optimum q*(Z;) depends on expectations 
computed with respect to the other factors q;(Z;) for i 4 j. We will therefore seek 
a consistent solution by first initializing all of the factors q;(Z;) appropriately and 
then cycling through the factors and replacing each in turn with a revised estimate 
given by the right-hand side of (10.9) evaluated using the current estimates for all of 
the other factors. Convergence is guaranteed because bound is convex with respect 
to each of the factors q;(Z;) (Boyd and Vandenberghe, 2004). 


10.1.2 Properties of factorized approximations 


Our approach to variational inference is based on a factorized approximation to 
the true posterior distribution. Let us consider for a moment the problem of approx- 
imating a general distribution by a factorized distribution. To begin with, we discuss 
the problem of approximating a Gaussian distribution using a factorized Gaussian, 
which will provide useful insight into the types of inaccuracy introduced in using 
factorized approximations. Consider a Gaussian distribution p(z) = N (z| u, A~') 
over two correlated variables z = (21, z2) in which the mean and precision have 


elements 
Ly Ay Aye 
= ; A= 10.10 
g (:) i ie ( ) 
and A21 = Aj2 due to the symmetry of the precision matrix. Now suppose we 


wish to approximate this distribution using a factorized Gaussian of the form q(z) = 
qi(21)q2(Z2). We first apply the general result (10.9) to find an expression for the 
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optimal factor qł (z1). In doing so it is useful to note that on the right-hand side we 
only need to retain those terms that have some functional dependence on zı because 
all other terms can be absorbed into the normalization constant. Thus we have 


Ingi (41) E- (In p(z)] + const 


; 1 
= Ez 5 (41 — p11)? Mai — (21 — Hi) At2(Z2 — H2) | + const 


1 ` 
-3A + zı Aa = z2 ( Efeza] = u2) + const. (10.11) 


Next we observe that the right-hand side of this expression is a quadratic function of 
zı, and so we can identify q* (z1) as a Gaussian distribution. It is worth emphasizing 
that we did not assume that g(z;) is Gaussian, but rather we derived this result by 
variational optimization of the KL divergence over all possible distributions q(z;). 
Note also that we do not need to consider the additive constant in (10.9) explicitly 
because it represents the normalization constant that can be found at the end by 
inspection if required. Using the technique of completing the square, we can identify 
the mean and precision of this Gaussian, giving 


(a) = N(zi|m, An) (10.12) 
where 
my = hi Ay Ai ( E[z2] = u2) š (10.13) 
By symmetry, g3 (z2) is also Gaussian and can be written as 
q3 (22) = N (z2|M2, Azz ) (10.14) 
in which 
mg = U2 — Ado Aoi (E[z1] = ui) . (10.15) 


Note that these solutions are coupled, so that q* (z1) depends on expectations com- 
puted with respect to q* (z2) and vice versa. In general, we address this by treating 
the variational solutions as re-estimation equations and cycling through the variables 
in turn updating them until some convergence criterion is satisfied. We shall see 
an example of this shortly. Here, however, we note that the problem is sufficiently 
simple that a closed form solution can be found. In particular, because E[z1] = m1 
and E|z2] = mz, we see that the two equations are satisfied if we take E[z;] = 4 
and E|z2] = u2, and it is easily shown that this is the only solution provided the dis- 
tribution is nonsingular. This result is illustrated in Figure 10.2(a). We see that the 
mean is correctly captured but that the variance of q(z) is controlled by the direction 
of smallest variance of p(z), and that the variance along the orthogonal direction is 
significantly under-estimated. It is a general result that a factorized variational ap- 
proximation tends to give approximations to the posterior distribution that are too 
compact. 

By way of comparison, suppose instead that we had been minimizing the reverse 
Kullback-Leibler divergence KL(p]||q). As we shall see, this form of KL divergence 
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Figure 10.2 Comparison of 
the two alternative forms for the 
Kullback-Leibler divergence. The 
green contours corresponding to 
1, 2, and 3 standard deviations for 
a correlated Gaussian distribution 
p(z) over two variables zı and 22, 
and the red contours represent 
the corresponding levels for an 
approximating distribution q(z) 
over the same variables given by 
the product of two independent 
univariate Gaussian distributions 
whose parameters are obtained by 
minimization of (a) the Kullback- 
Leibler divergence KL(q||p), and 
(b) the reverse Kullback-Leibler 
divergence KL(pl|q). 
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is used in an alternative approximate inference framework called expectation prop- 
agation. We therefore consider the general problem of minimizing KL(p||q) when 
q(Z) is a factorized approximation of the form (10.5). The KL divergence can then 
be written in the form 


M 
KL(p||q) = - f2 [matz dZ + const (10.16) 
4=1 


where the constant term is simply the entropy of p(Z) and so does not depend on 
q(Z). We can now optimize with respect to each of the factors q;(Z,;), which is 
easily done using a Lagrange multiplier to give 


qj (Z;) = feall dZ; = p(Z;). (10.17) 


tAj 


In this case, we find that the optimal solution for q;(Z;) is just given by the corre- 
sponding marginal distribution of p(Z). Note that this is a closed-form solution and 
so does not require iteration. 

To apply this result to the illustrative example of a Gaussian distribution p(z) 
over a vector z we can use (2.98), which gives the result shown in Figure 10.2(b). 
We see that once again the mean of the approximation is correct, but that it places 
significant probability mass in regions of variable space that have very low probabil- 
ity. 

The difference between these two results can be understood by noting that there 
is a large positive contribution to the Kullback-Leibler divergence 


KL(q||p) = - f a2 In {am} dZ (10.18) 
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Figure 10.3 Another comparison of the two alternative forms for the Kullback-Leibler divergence. (a) The blue 
contours show a bimodal distribution p(Z) given by a mixture of two Gaussians, and the red contours correspond 
to the single Gaussian distribution q(Z) that best approximates p(Z) in the sense of minimizing the Kullback- 
Leibler divergence KL(p||q). (b) As in (a) but now the red contours correspond to a Gaussian distribution q(Z) 
found by numerical minimization of the Kullback-Leibler divergence KL(q||p). (c) As in (b) but showing a different 
local minimum of the Kullback-Leibler divergence. 
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from regions of Z space in which p(Z) is near zero unless q(Z) is also close to 
zero. Thus minimizing this form of KL divergence leads to distributions q(Z) that 
avoid regions in which p(Z) is small. Conversely, the Kullback-Leibler divergence 
KL(p]|q) is minimized by distributions q(Z) that are nonzero in regions where p(Z) 
is nonzero. 

We can gain further insight into the different behaviour of the two KL diver- 
gences if we consider approximating a multimodal distribution by a unimodal one, 
as illustrated in Figure 10.3. In practical applications, the true posterior distri- 
bution will often be multimodal, with most of the posterior mass concentrated in 
some number of relatively small regions of parameter space. These multiple modes 
may arise through nonidentifiability in the latent space or through complex nonlin- 
ear dependence on the parameters. Both types of multimodality were encountered in 
Chapter 9 in the context of Gaussian mixtures, where they manifested themselves as 
multiple maxima in the likelihood function, and a variational treatment based on the 
minimization of KL(q||p) will tend to find one of these modes. By contrast, if we 
were to minimize KL(p||q), the resulting approximations would average across all 
of the modes and, in the context of the mixture model, would lead to poor predictive 
distributions (because the average of two good parameter values is typically itself 
not a good parameter value). It is possible to make use of KL(p]|q) to define a useful 
inference procedure, but this requires a rather different approach to the one discussed 
here, and will be considered in detail when we discuss expectation propagation. 

The two forms of Kullback-Leibler divergence are members of the alpha family 
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of divergences (Ali and Silvey, 1966; Amari, 1985; Minka, 2005) defined by 


Da(plla) = = (1 i f pla) OFM q(x)0 9/2 ar) (10.19) 


where —oo < a < oo is a continuous parameter. The Kullback-Leibler divergence 
KL(p||q) corresponds to the limit a — 1, whereas KL(q||p) corresponds to the limit 
a — —1. For all values of a we have Da(p||q) > 0, with equality if, and only if, 
p(x) = q(x). Suppose p(x) is a fixed distribution, and we minimize Da(p||q) with 
respect to some set of distributions g(x). Then for a < —1 the divergence is zero 
forcing, so that any values of x for which p(x) = 0 will have q(x) = 0, and typically 
q(x) will under-estimate the support of p(x) and will tend to seek the mode with the 
largest mass. Conversely for œ > 1 the divergence is zero-avoiding, so that values 
of x for which p(x) > 0 will have g(a) > 0, and typically q(x) will stretch to cover 
all of p(x), and will over-estimate the support of p(x). When a = 0 we obtain a 
symmetric divergence that is linearly related to the Hellinger distance given by 


Du(pllq) = f (p(a)'/? — q(a)'/?) dex. (10.20) 


The square root of the Hellinger distance is a valid distance metric. 


10.1.3 Example: The univariate Gaussian 


We now illustrate the factorized variational approximation using a Gaussian dis- 
tribution over a single variable x (MacKay, 2003). Our goal is to infer the posterior 
distribution for the mean u and precision 7, given a data set D = {x,,...,xy} of 
observed values of x which are assumed to be drawn independently from the Gaus- 
sian. The likelihood function is given by 


r \N/2 eel ‘ 
pDl r) = (5) © exp) -2 Sen n}. (10.21) 
We now introduce conjugate prior distributions for u and T given by 


P(ult) = N (ult, (Aor) ~*) (10.22) 
p(t) = Gam(r|ao, bo) (10.23) 


where Gam(rT|do, bo) is the gamma distribution defined by (2.146). Together these 
distributions constitute a Gaussian-Gamma conjugate prior distribution. 

For this simple problem the posterior distribution can be found exactly, and again 
takes the form of a Gaussian-gamma distribution. However, for tutorial purposes 
we will consider a factorized variational approximation to the posterior distribution 
given by 

a(u, T) = qu (u)ar (7). (10.24) 
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Note that the true posterior distribution does not factorize in this way. The optimum 
factors q,,(j4) and q-(7) can be obtained from the general result (10.9) as follows. 
For q, (u) we have 


Ingi(u) = E,[lnp(D|u, T) + In p(plr)] + const 


Er] 


N 
ne {ut — uo)? + >, C nk + const. (10.25) 


Completing the square over u we see that q,,(j) is a Gaussian M (ulu N; Ay ) with 
mean and precision given by 


Aoko + NT 
= ee 10.26 
HN Xo HN ( ) 
AN = (Ao +N) E[r]. (10.27) 


Note that for N — oo this gives the maximum likelihood result in which wy = ¥ 
and the precision is infinite. 
Similarly, the optimal solution for the factor q, (7) is given by 


Ingi(7) = E, [Inp(P|py,7) +n p(ulr)] + In p(T) + const 


N 
= (a)—1)Int— bot + zT 


T 


N 
—-E,, Sot UY + Ao(u nr + const (10.28) 


2 


n=1 


and hence q,(7T) is a gamma distribution Gam(r|ay,, by) with parameters 


N 

an = wt (10.29) 
1 N 

by = bot 5By [Ze — u)? +ou — m! . (10.30) 


Again this exhibits the expected behaviour when N — oo. 

It should be emphasized that we did not assume these specific functional forms 
for the optimal distributions q, (4) and q-(7). They arose naturally from the structure 
of the likelihood function and the corresponding conjugate priors. 

Thus we have expressions for the optimal distributions q, (4) and g,(7) each of 
which depends on moments evaluated with respect to the other distribution. One ap- 
proach to finding a solution is therefore to make an initial guess for, say, the moment 
E[r] and use this to re-compute the distribution q,,(). Given this revised distri- 
bution we can then extract the required moments E[u] and E[j.?], and use these to 
recompute the distribution g,-(7), and so on. Since the space of hidden variables for 
this example is only two dimensional, we can illustrate the variational approxima- 
tion to the posterior distribution by plotting contours of both the true posterior and 
the factorized approximation, as illustrated in Figure 10.4. 
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Figure 10.4 Illustration of variational inference for the mean x and precision 7 of a univariate Gaussian distribu- 
tion. Contours of the true posterior distribution p(:,7|D) are shown in green. (a) Contours of the initial factorized 
approximation q„(u)q-(T) are shown in blue. (b) After re-estimating the factor q,,(w). (c) After re-estimating the 
factor q,(7). (d) Contours of the optimal factorized approximation, to which the iterative scheme converges, are 
shown in red. 


In general, we will need to use an iterative approach such as this in order to 
solve for the optimal factorized posterior distribution. For the very simple example 
we are considering here, however, we can find an explicit solution by solving the 
simultaneous equations for the optimal factors q„ (4) and q(T). Before doing this, 
we can simplify these expressions by considering broad, noninformative priors in 
which uo = ao = bo = Ao = 0. Although these parameter settings correspond to 
improper priors, we see that the posterior distribution is still well defined. Using the 

Appendix B standard result E[r] = ay /by for the mean of a gamma distribution, together with 
(10.29) and (10.30), we have 


1 What a ae 
Ja =E | ag DL (ta p)? | = 2? — 228 ly] + Elp’). (10.31) 
n=1 


E[r] 


Then, using (10.26) and (10.27), we obtain the first and second order moments of 
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qu (u) in the form 


E(u] = 2, Elu?] = z? + (10.32) 


NE|7] 


We can now substitute these moments into (10.31) and then solve for E[r] to give 


1 1 = | 
i Nai ~*) 
1 N 
= $m- (10.33) 
n=1 


We recognize the right-hand side as the familiar unbiased estimator for the variance 
of a univariate Gaussian distribution, and so we see that the use of a Bayesian ap- 
proach has avoided the bias of the maximum likelihood solution. 


10.1.4 Model comparison 


As well as performing inference over the hidden variables Z, we may also 
wish to compare a set of candidate models, labelled by the index m, and having 
prior probabilities p(m). Our goal is then to approximate the posterior probabilities 
p(m|X), where X is the observed data. This is a slightly more complex situation 
than that considered so far because different models may have different structure 
and indeed different dimensionality for the hidden variables Z. We cannot there- 
fore simply consider a factorized approximation q(Z)q(m), but must instead recog- 
nize that the posterior over Z must be conditioned on m, and so we must consider 
q(Z,m) = q(Z|m)q(m). We can readily verify the following decomposition based 
on this variational distribution 


In p(X) = Lm — `> > q(Z|m)q(m) In {ean (10.34) 


where the £m is a lower bound on In p(X) and is given by 


C= DY atzim)aten) inf A I (10.35) 


Here we are assuming discrete Z, but the same analysis applies to continuous latent 
variables provided the summations are replaced with integrations. We can maximize 
Lm With respect to the distribution g(m) using a Lagrange multiplier, with the result 


q(m) x plm) exp{Ly}. (10.36) 


However, if we maximize Lm with respect to the q(Z|m), we find that the solutions 
for different m are coupled, as we expect because they are conditioned on m. We 
proceed instead by first optimizing each of the q(Z|m) individually by optimization 
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of (10.35), and then subsequently determining the q(m) using (10.36). After nor- 
malization the resulting values for g(m) can be used for model selection or model 
averaging in the usual way. 


Illustration: Variational Mixture of Gaussians 


We now return to our discussion of the Gaussian mixture model and apply the vari- 
ational inference machinery developed in the previous section. This will provide a 
good illustration of the application of variational methods and will also demonstrate 
how a Bayesian treatment elegantly resolves many of the difficulties associated with 
the maximum likelihood approach (Attias, 1999b). The reader is encouraged to work 
through this example in detail as it provides many insights into the practical appli- 
cation of variational methods. Many Bayesian models, corresponding to much more 
sophisticated distributions, can be solved by straightforward extensions and general- 
izations of this analysis. 

Our starting point is the likelihood function for the Gaussian mixture model, il- 
lustrated by the graphical model in Figure 9.6. For each observation x,, we have 
a corresponding latent variable z,, comprising a l-of-K binary vector with ele- 
ments Znķ for k = 1,..., K. As before we denote the observed data set by X = 
{x1,...,X}, and similarly we denote the latent variables by Z = {z1,..., zy}. 
From (9.10) we can write down the conditional distribution of Z, given the mixing 
coefficients 7r, in the form 


N K 
p(Z\r) = | [|] [7 (10.37) 


Similarly, from (9.11), we can write down the conditional distribution of the ob- 
served data vectors, given the latent variables and the component parameters 


N K 
p(XIZ, u, A) = TT | [NV (Snu Ag) (10.38) 


n=1k=1 


where u = {up} and A = {Ax}. Note that we are working in terms of precision 
matrices rather than covariance matrices as this somewhat simplifies the mathemat- 
ics. 

Next we introduce priors over the parameters u, A and m. The analysis is con- 
siderably simplified if we use conjugate prior distributions. We therefore choose a 
Dirichlet distribution over the mixing coefficients 7 


K 
p(T) = Dir(rlao) = C(ao) | | mer (10.39) 
k=1 


where by symmetry we have chosen the same parameter a for each of the compo- 
nents, and C (œo) is the normalization constant for the Dirichlet distribution defined 


Figure 10.5 Directed acyclic graph representing the Bayesian mix- 
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ture of Gaussians model, in which the box (plate) de- 
notes a set of N i.i.d. observations. Here u denotes 
{u} and A denotes {A} }. 


by (B.23). As we have seen, the parameter a can be interpreted as the effective 
prior number of observations associated with each component of the mixture. If the 
value of œo is small, then the posterior distribution will be influenced primarily by 
the data rather than by the prior. 

Similarly, we introduce an independent Gaussian-Wishart prior governing the 
mean and precision of each Gaussian component, given by 


plu, A) = p(m|A)p(A) 


K 
= [IV (mmo, (80A) ) WAkWo, vo) (10.40) 


k=1 


because this represents the conjugate prior distribution when both the mean and pre- 
cision are unknown. Typically we would choose mp = 0 by symmetry. 

The resulting model can be represented as a directed graph as shown in Fig- 
ure 10.5. Note that there is a link from A to u since the variance of the distribution 
over u in (10.40) is a function of A. 

This example provides a nice illustration of the distinction between latent vari- 
ables and parameters. Variables such as z,, that appear inside the plate are regarded 
as latent variables because the number of such variables grows with the size of the 
data set. By contrast, variables such as ys that are outside the plate are fixed in 
number independently of the size of the data set, and so are regarded as parameters. 
From the perspective of graphical models, however, there is really no fundamental 
difference between them. 


10.2.1 Variational distribution 


In order to formulate a variational treatment of this model, we next write down 
the joint distribution of all of the random variables, which is given by 


P(X, Z, T, p, A) = p(X|Z, p, A)p(Z|m) p(w) p(w A pA) (10.41) 


in which the various factors are defined above. The reader should take a moment to 
verify that this decomposition does indeed correspond to the probabilistic graphical 
model shown in Figure 10.5. Note that only the variables X = {x1,..., Xy} are 
observed. 


476 10. APPROXIMATE INFERENCE 


Exercise 10.12 


We now consider a variational distribution which factorizes between the latent 
variables and the parameters so that 


q(Z, T, p, A) = q(Z)a(m, p, A). (10.42) 


It is remarkable that this is the only assumption that we need to make in order to 
obtain a tractable practical solution to our Bayesian mixture model. In particular, the 
functional form of the factors q(Z) and q(7, p, A) will be determined automatically 
by optimization of the variational distribution. Note that we are omitting the sub- 
scripts on the q distributions, much as we do with the p distributions in (10.41), and 
are relying on the arguments to distinguish the different distributions. 

The corresponding sequential update equations for these factors can be easily 
derived by making use of the general result (10.9). Let us consider the derivation of 
the update equation for the factor q(Z). The log of the optimized factor is given by 


In q*(Z) = Er ya [In p(X, Z, m, u, A)|] + const. (10.43) 


We now make use of the decomposition (10.41). Note that we are only interested in 
the functional dependence of the right-hand side on the variable Z. Thus any terms 
that do not depend on Z can be absorbed into the additive normalization constant, 
giving 


Ing*(Z) =E, [In p(Z|7)| + E,, a [In p(X|Z, u, A)| + const. (10.44) 


Substituting for the two conditional distributions on the right-hand side, and again 
absorbing any terms that are independent of Z into the additive constant, we have 


N K 
Ing*(Z) = $ X nk In png + const (10.45) 


n=1 k=1 


where we have defined 


1 D 
lnpnk = Ellnzg] + Fi E (In |Ax|] — T ln(27) 
l 
FE py Ay [n Aln] 01046) 


where D is the dimensionality of the data variable x. Taking the exponential of both 
sides of (10.45) we obtain 


N K 
(z) x TT J J ot. (10.47) 


n=1k=1 


Requiring that this distribution be normalized, and noting that for each value of n 
the quantities zn% are binary and sum to 1 over all values of k, we obtain 


N K 
a*(Z) = | J [ [ee (10.48) 


n=1k=1 


10.2. Illustration: Variational Mixture of Gaussians 477 


where 
Pnk 


K 


‘> Pnj 


j=l 


(10.49) 


Tnk = 


We see that the optimal solution for the factor q(Z) takes the same functional form 

as the prior p(Z|7). Note that because pny is given by the exponential of a real 

quantity, the quantities r,,;, will be nonnegative and will sum to one, as required. 
For the discrete distribution g*(Z) we have the standard result 


E | Znk] = Tnk (10.50) 


from which we see that the quantities rng are playing the role of responsibilities. 
Note that the optimal solution for q*(Z) depends on moments evaluated with respect 
to the distributions of other variables, and so again the variational update equations 
are coupled and must be solved iteratively. 

At this point, we shall find it convenient to define three statistics of the observed 
data set evaluated with respect to the responsibilities, given by 


Nk = Da (10.51) 
7 n 
x = Mi >, TnkXn (10.52) 
1 Š = = \T 
Sk = N. 3 Tnk(Xn — Xk)(Xn — Xk). (10.53) 


Note that these are analogous to quantities evaluated in the maximum likelihood EM 
algorithm for the Gaussian mixture model. 

Now let us consider the factor g(a, p, A) in the variational posterior distribu- 
tion. Again using the general result (10.9) we have 


K 
Ing*(m, p, A) =Inp(r) + S— Inp(u,, Ax) + Ez [In p(Z|7)] 
k=1 
K N 
+X X Elen] nN (Xn |p, Ag") + const. (10.54) 


k=1 n=1 


We observe that the right-hand side of this expression decomposes into a sum of 
terms involving only 7 together with terms only involving p and A, which implies 
that the variational posterior q(z, u, A) factorizes to give q(7)q(u, A). Further- 
more, the terms involving yz and A themselves comprise a sum over k of terms 
involving p, and Ax leading to the further factorization 


K 
q(T, p, A) = a() [J alur A). (10.55) 
k=1 
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Identifying the terms on the right-hand side of (10.54) that depend on zr, we have 


K K N 
In g* (m) = (ao — 1) i: Tk + Se So rnd ln Tk + const (10.56) 
k=1 


k=1 n=1 


where we have used (10.50). Taking the exponential of both sides, we recognize 
q* (7) as a Dirichlet distribution 


(r) = Dir(r|a) (10.57) 
where œ has components a, given by 
Qk = Qo + Nk. (10.58) 


Finally, the variational posterior distribution q* (4%, Ax) does not factorize into 
the product of the marginals, but we can always use the product rule to write it in the 
form q* (Hp, Ax) = q*(Hg|Ak)q* (Ax). The two factors can be found by inspecting 
(10.54) and reading off those terms that involve u, and Ax. The result, as expected, 
is a Gaussian-Wishart distribution and is given by 


(Mas Ak) =N (wg lime, (BeAr) >) WAW, ve) (10.59) 
where we have defined 
Be = BotNe (10.60) 
m, = z (omo + Nit) (10.61) 
Wo! = Wt + N:S 4 “AS (Zr —mo)(X, — mo)" (10.62) 
Ve = Vot Ng. (10.63) 


These update equations are analogous to the M-step equations of the EM algorithm 
for the maximum likelihood solution of the mixture of Gaussians. We see that the 
computations that must be performed in order to update the variational posterior 
distribution over the model parameters involve evaluation of the same sums over the 
data set, as arose in the maximum likelihood treatment. 

In order to perform this variational M step, we need the expectations E[z,,,] = 
Tnk representing the responsibilities. These are obtained by normalizing the pnw that 
are given by (10.46). We see that this expression involves expectations with respect 
to the variational distributions of the parameters, and these are easily evaluated to 
give 


Eup, Ak (xn E Ug) Ane = L,)| 
DB, + ¥e(Xn — Me) We(Xn — mk) (10.64) 


D 5 
yo (A) t Din2+In|W,| (10.65) 


(10.66) 


In A, = E [ln |Ak] 


I 
= 
R 
= 

| 
€ 
© 


ln Tk = E fln mk] 
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where we have introduced definitions of A, and Tg, and 7)(-) is the digamma function 
defined by (B.25), with a = X @p. The results (10.65) and (10.66) follow from 
the standard properties of the Wishart and Dirichlet distributions. 

If we substitute (10.64), (10.65), and (10.66) into (10.46) and make use of 
(10.49), we obtain the following result for the responsibilities 


D Vk 
20, 2 


Notice the similarity to the corresponding result for the responsibilities in maximum 
likelihood EM, which from (9.13) can be written in the form 


Tnk X TA? exp { (Xa m) Wi (Xn — mo} . (10.67) 


1 
Tak X Tk|Ap|!? exp f -ien — pty) Ak (Xn — m)} (10.68) 


where we have used the precision in place of the covariance to highlight the similarity 
to (10.67). 

Thus the optimization of the variational posterior distribution involves cycling 
between two stages analogous to the E and M steps of the maximum likelihood EM 
algorithm. In the variational equivalent of the E step, we use the current distributions 
over the model parameters to evaluate the moments in (10.64), (10.65), and (10.66) 
and hence evaluate Efznk] = rnk. Then in the subsequent variational equivalent 
of the M step, we keep these responsibilities fixed and use them to re-compute the 
variational distribution over the parameters using (10.57) and (10.59). In each case, 
we see that the variational posterior distribution has the same functional form as the 
corresponding factor in the joint distribution (10.41). This is a general result and is 
a consequence of the choice of conjugate distributions. 

Figure 10.6 shows the results of applying this approach to the rescaled Old Faith- 
ful data set for a Gaussian mixture model having K = 6 components. We see that 
after convergence, there are only two components for which the expected values 
of the mixing coefficients are numerically distinguishable from their prior values. 
This effect can be understood qualitatively in terms of the automatic trade-off in a 
Bayesian model between fitting the data and the complexity of the model, in which 
the complexity penalty arises from components whose parameters are pushed away 
from their prior values. Components that take essentially no responsibility for ex- 
plaining the data points have r,, œ~ 0 and hence Nọ œ 0. From (10.58), we see 
that a, ~ Qo and from (10.60)-(10.63) we see that the other parameters revert to 
their prior values. In principle such components are fitted slightly to the data points, 
but for broad priors this effect is too small to be seen numerically. For the varia- 
tional Gaussian mixture model the expected values of the mixing coefficients in the 
posterior distribution are given by 


: an + Nz 
ie, 10. 
[7x] Kay +N (10.63) 


Consider a component for which N;, ~ 0 and a; œ ao. If the prior is broad so that 
ao — 0, then E[7;,] — 0 and the component plays no role in the model, whereas if 
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Figure 10.6 Variational Bayesian 
mixture of K = 6 Gaussians ap- 
plied to the Old Faithful data set, in 
which the ellipses denote the one 
standard-deviation density contours 
for each of the components, and the 
density of red ink inside each ellipse 
corresponds to the mean value of 
the mixing coefficient for each com- 
ponent. The number in the top left 
of each diagram shows the num- 
ber of iterations of variational infer- 
ence. Components whose expected 
mixing coefficient are numerically in- 
distinguishable from zero are not 
plotted. 


the prior tightly constrains the mixing coefficients so that ag — oo, then Efry] > 
1/K. 

In Figure 10.6, the prior over the mixing coefficients is a Dirichlet of the form 
(10.39). Recall from Figure 2.5 that for ag < 1 the prior favours solutions in which 
some of the mixing coefficients are zero. Figure 10.6 was obtained using a = 107, 
and resulted in two components having nonzero mixing coefficients. If instead we 
choose œo = 1 we obtain three components with nonzero mixing coefficients, and 
for œ = 10 all six components have nonzero mixing coefficients. 

As we have seen there is a close similarity between the variational solution for 
the Bayesian mixture of Gaussians and the EM algorithm for maximum likelihood. 
In fact if we consider the limit N — oo then the Bayesian treatment converges to the 
maximum likelihood EM algorithm. For anything other than very small data sets, 
the dominant computational cost of the variational algorithm for Gaussian mixtures 
arises from the evaluation of the responsibilities, together with the evaluation and 
inversion of the weighted data covariance matrices. These computations mirror pre- 
cisely those that arise in the maximum likelihood EM algorithm, and so there is little 
computational overhead in using this Bayesian approach as compared to the tradi- 
tional maximum likelihood one. There are, however, some substantial advantages. 
First of all, the singularities that arise in maximum likelihood when a Gaussian com- 
ponent ‘collapses’ onto a specific data point are absent in the Bayesian treatment. 
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Indeed, these singularities are removed if we simply introduce a prior and then use a 
MAP estimate instead of maximum likelihood. Furthermore, there is no over-fitting 
if we choose a large number K of components in the mixture, as we saw in Fig- 
ure 10.6. Finally, the variational treatment opens up the possibility of determining 
the optimal number of components in the mixture without resorting to techniques 
such as cross validation. 


10.2.2 Variational lower bound 


We can also straightforwardly evaluate the lower bound (10.3) for this model. 
In practice, it is useful to be able to monitor the bound during the re-estimation in 
order to test for convergence. It can also provide a valuable check on both the math- 
ematical expressions for the solutions and their software implementation, because at 
each step of the iterative re-estimation procedure the value of this bound should not 
decrease. We can take this a stage further to provide a deeper test of the correctness 
of both the mathematical derivation of the update equations and of their software im- 
plementation by using finite differences to check that each update does indeed give 
a (constrained) maximum of the bound (Svensén and Bishop, 2004). 

For the variational mixture of Gaussians, the lower bound (10.3) is given by 


— p(X, Z, 7, u, A) 
L = © J| zramm] ARAN } amduan 


= Ejlnp(X, Z, 7, u, A)| — Eln q(Z, m, u, A)] 
= Ell p(X|Z, n, A)] + Eln p(Z|7)] + Elin p(m)] + Eln p(y, A)] 
—E[In 4(Z)] — E[In q(7)] — E[n q(x, A)] (10.70) 


where, to keep the notation uncluttered, we have omitted the x superscript on the 
q distributions, along with the subscripts on the expectation operators because each 
expectation is taken with respect to all of the random variables in its argument. The 
various terms in the bound are easily evaluated to give the following results 


K 

‘ 1 < = 

E[In p(X|Z, p, A)| = z > Nk { In Ak — DE} — 1, Tr(S,We) 
k=1 


=v, (Xp_ — my)" W; (Xk — mg) — Din(2n)} (10.71) 


N K 
Elnp(Zir)] = dd Tnk In Ftp (10.72) 
n=1 k=1 


K 
Ellnp(r)] = InC(ao) + (ao — 1) X` Inf (10.73) 
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_ Pbo 
Be 


K 
1 
Ellin p(w, A)|] = N= ad, {pin (80/27) +InA 


—BoV_ (my = mo) W; (m, = mo) + Klin B(Wo, Vo) 


Aaa . i 
+ X nå- 5 S 2 ve Tr(Wo 'Wx) (10.74) 
k=1 k=1 
N K 
Ellin q(Z)] = 2 nk IN Tnk (10.75) 
n=1 k=1 
K 
Bln g(m)] = X (ax —1)In7t, +n C(@) (10.76) 
k=1 
X (1 D (6 D 
` ae k 
Ellin g(u, A)|] = deh Ps In (=) 3 Hla(As)]} (10.77) 


where D is the dimensionality of x, H[q(A,)] is the entropy of the Wishart distribu- 
tion given by (B.82), and the coefficients C (œ) and B(W, v) are defined by (B.23) 
and (B.79), respectively. Note that the terms involving expectations of the logs of the 
q distributions simply represent the negative entropies of those distributions. Some 
simplifications and combination of terms can be performed when these expressions 
are summed to give the lower bound. However, we have kept the expressions sepa- 
rate for ease of understanding. 

Finally, it is worth noting that the lower bound provides an alternative approach 
for deriving the variational re-estimation equations obtained in Section 10.2.1. To do 
this we use the fact that, since the model has conjugate priors, the functional form of 
the factors in the variational posterior distribution is known, namely discrete for Z, 
Dirichlet for 7, and Gaussian-Wishart for (ug, Ax). By taking general parametric 
forms for these distributions we can derive the form of the lower bound as a function 
of the parameters of the distributions. Maximizing the bound with respect to these 
parameters then gives the required re-estimation equations. 


10.2.3 Predictive density 


In applications of the Bayesian mixture of Gaussians model we will often be 
interested in the predictive density for a new value X of the observed variable. As- 
sociated with this observation will be a corresponding latent variable Z, and the pre- 
dictive density is then given by 


p(x|X) = X / ff (|Z, u, A)p(Z\m)p(ar, u, A| X)drdudA (10.78) 
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where p(7r, u, A|X) is the (unknown) true posterior distribution of the parameters. 
Using (10.37) and (10.38) we can first perform the summation over Z to give 


K 
p(x|X) = S TN (|My, Az) p(w, u, A|X) dr du dA. (10.79) 
k=1 


Because the remaining integrations are intractable, we approximate the predictive 
density by replacing the true posterior distribution p(7r, p, A|X) with its variational 
approximation q(7)q(jz, A) to give 


K 
p(x|X) = ei TN (Xl oy, Ag) (T)ar, An) dr dpe, Ax (10.80) 
k=1 


where we have made use of the factorization (10.55) and in each term we have im- 
plicitly integrated out all variables { Hj, A j} for j # k The remaining integrations 
can now be evaluated analytically giving a mixture of Student’s t-distributions 


K 
A 1 A 

X)==) ast Lp at 10.81 

p(x|X) a2 apSt(x|my, Lk, Vk + ) ( ) 


in which the kt component has mean mx, and the precision is given by 
(ve +1—D) Bp 
(1+ Bx) 


in which vx is given by (10.63). When the size N of the data set is large the predictive 
distribution (10.81) reduces to a mixture of Gaussians. 


Ly = Wi, (10.82) 


10.2.4 Determining the number of components 


We have seen that the variational lower bound can be used to determine a pos- 
terior distribution over the number K of components in the mixture model. There 
is, however, one subtlety that needs to be addressed. For any given setting of the 
parameters in a Gaussian mixture model (except for specific degenerate settings), 
there will exist other parameter settings for which the density over the observed vari- 
ables will be identical. These parameter values differ only through a re-labelling of 
the components. For instance, consider a mixture of two Gaussians and a single ob- 
served variable x, in which the parameters have the values 7; = a, 72 = b, 41 = C, 
u2 = d, 0, = e, 02 = f. Then the parameter values 7, b, T2 a, Hı d, 
[lg = c, 01 = f, o2 = e, in which the two components have been exchanged, will 
by symmetry give rise to the same value of p(x). If we have a mixture model com- 
prising K components, then each parameter setting will be a member of a family of 
K! equivalent settings. 

In the context of maximum likelihood, this redundancy is irrelevant because the 
parameter optimization algorithm (for example EM) will, depending on the initial- 
ization of the parameters, find one specific solution, and the other equivalent solu- 
tions play no role. In a Bayesian setting, however, we marginalize over all possible 
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Figure 10.7 Plot of the variational lower bound r r r r : r 
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£ versus the number K of com- 
ponents in the Gaussian mixture 
model, for the Old Faithful data, 
showing a distinct peak at K = 
2 components. For each value 
of K, the model is trained from 4h 

100 different random starts, and HF- w 
the results shown as ‘+’ symbols p(DIK) 
plotted with small random hori- m- 
zontal perturbations so that they 
can be distinguished. Note that 
some solutions find suboptimal 
local maxima, but that this hap- 
pens infrequently. 


parameter values. We have seen in Figure 10.2 that if the true posterior distribution 
is multimodal, variational inference based on the minimization of KL(q||p) will tend 
to approximate the distribution in the neighbourhood of one of the modes and ignore 
the others. Again, because equivalent modes have equivalent predictive densities, 
this is of no concern provided we are considering a model having a specific number 
K of components. If, however, we wish to compare different values of K, then we 
need to take account of this multimodality. A simple approximate solution is to add 
a term In K! onto the lower bound when used for model comparison and averaging. 
Figure 10.7 shows a plot of the lower bound, including the multimodality fac- 
tor, versus the number K of components for the Old Faithful data set. It is worth 
emphasizing once again that maximum likelihood would lead to values of the likeli- 
hood function that increase monotonically with K (assuming the singular solutions 
have been avoided, and discounting the effects of local maxima) and so cannot be 
used to determine an appropriate model complexity. By contrast, Bayesian inference 
automatically makes the trade-off between model complexity and fitting the data. 
This approach to the determination of K requires that a range of models having 
different K values be trained and compared. An alternative approach to determining 
a suitable value for K is to treat the mixing coefficients m as parameters and make 
point estimates of their values by maximizing the lower bound (Corduneanu and 
Bishop, 2001) with respect to m instead of maintaining a probability distribution 
over them as in the fully Bayesian approach. This leads to the re-estimation equation 


1 N 
Th= 7 >, Tnk (10.83) 


and this maximization is interleaved with the variational updates for the q distribution 
over the remaining parameters. Components that provide insufficient contribution 
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to explaining the data will have their mixing coefficients driven to zero during the 
optimization, and so they are effectively removed from the model through automatic 
relevance determination. This allows us to make a single training run in which we 
start with a relatively large initial value of K, and allow surplus components to be 
pruned out of the model. The origins of the sparsity when optimizing with respect to 
hyperparameters is discussed in detail in the context of the relevance vector machine. 


10.2.5 Induced factorizations 


In deriving these variational update equations for the Gaussian mixture model, 
we assumed a particular factorization of the variational posterior distribution given 
by (10.42). However, the optimal solutions for the various factors exhibit additional 
factorizations. In particular, the solution for q*( u, A) is given by the product of an 
independent distribution q* (up, Ax) over each of the components k of the mixture, 
whereas the variational posterior distribution q*(Z) over the latent variables, given 
by (10.48), factorizes into an independent distribution q*(z,,) for each observation n 
(note that it does not further factorize with respect to k because, for each value of n, 
the z,% are constrained to sum to one over k). These additional factorizations are a 
consequence of the interaction between the assumed factorization and the conditional 
independence properties of the true distribution, as characterized by the directed 
graph in Figure 10.5. 

We shall refer to these additional factorizations as induced factorizations be- 
cause they arise from an interaction between the factorization assumed in the varia- 
tional posterior distribution and the conditional independence properties of the true 
joint distribution. In a numerical implementation of the variational approach it is 
important to take account of such additional factorizations. For instance, it would 
be very inefficient to maintain a full precision matrix for the Gaussian distribution 
over a set of variables if the optimal form for that distribution always had a diago- 
nal precision matrix (corresponding to a factorization with respect to the individual 
variables described by that Gaussian). 

Such induced factorizations can easily be detected using a simple graphical test 
based on d-separation as follows. We partition the latent variables into three disjoint 
groups A, B, C and then let us suppose that we are assuming a factorization between 
C and the remaining latent variables, so that 


q(A, B,C) = q(A, B)q(C). (10.84) 


Using the general result (10.9), together with the product rule for probabilities, we 
see that the optimal solution for g(A, B) is given by 
Ing*(A,B) = Ecllnp(X, A,B, C)] + const 
= Eo{Inp(A, B|X, C)] + const. (10.85) 


We now ask whether this resulting solution will factorize between A and B, in 
other words whether q*(A, B) = q*(A)q*(B). This will happen if, and only if, 
In p(A, B]X, C) = Inp(A|X, C) + In p(B|X, C), that is, if the conditional inde- 
pendence relation 

A iL B|X,C (10.86) 
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is satisfied. We can test to see if this relation does hold, for any choice of A and B 
by making use of the d-separation criterion. 

To illustrate this, consider again the Bayesian mixture of Gaussians represented 
by the directed graph in Figure 10.5, in which we are assuming a variational fac- 
torization given by (10.42). We can see immediately that the variational posterior 
distribution over the parameters must factorize between 7 and the remaining param- 
eters yz and A because all paths connecting 7 to either u or A must pass through 
one of the nodes z,, all of which are in the conditioning set for our conditional inde- 
pendence test and all of which are head-to-tail with respect to such paths. 


Variational Linear Regression 


As a second illustration of variational inference, we return to the Bayesian linear 
regression model of Section 3.3. In the evidence framework, we approximated the 
integration over a and ( by making point estimates obtained by maximizing the log 
marginal likelihood. A fully Bayesian approach would integrate over the hyperpa- 
rameters as well as over the parameters. Although exact integration is intractable, 
we can use variational methods to find a tractable approximation. In order to sim- 
plify the discussion, we shall suppose that the noise precision parameter 8 is known, 
and is fixed to its true value, although the framework is easily extended to include 
the distribution over Ø. For the linear regression model, the variational treatment 
will turn out to be equivalent to the evidence framework. Nevertheless, it provides a 
good exercise in the use of variational methods and will also lay the foundation for 
variational treatment of Bayesian logistic regression in Section 10.6. 
Recall that the likelihood function for w, and the prior over w, are given by 


N 

ptlw) = [Niwon 871) (10.87) 
n=1 

p(wla) = N(w|0,a7"I) (10.88) 


where @,, = $(X,,). We now introduce a prior distribution over a. From our dis- 
cussion in Section 2.3.6, we know that the conjugate prior for the precision of a 
Gaussian is given by a gamma distribution, and so we choose 


pla) = Gam(alao, bo) (10.89) 


where Gam(-|-, -) is defined by (B.26). Thus the joint distribution of all the variables 
is given by 
p(t, w, a) = p(t|w)p(wla)p(a). (10.90) 


This can be represented as a directed graphical model as shown in Figure 10.8. 


10.3.1 Variational distribution 


Our first goal is to find an approximation to the posterior distribution p(w, at). 
To do this, we employ the variational framework of Section 10.1, with a variational 
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Figure 10.8 Probabilistic graphical model representing the joint dis- 
tribution (10.90) for the Bayesian linear regression 
model. 


posterior distribution given by the factorized expression 


q(w, a) = q(w)q(a). (10.91) 


We can find re-estimation equations for the factors in this distribution by making use 
of the general result (10.9). Recall that for each factor, we take the log of the joint 
distribution over all variables and then average with respect to those variables not in 
that factor. Consider first the distribution over a. Keeping only terms that have a 
functional dependence on a, we have 


Ing*(a) = Inp(a) + Ey [In p(wla)] + const 
M 
= (a) —1)lna-— boa A Ina E[w'w]+ const. (10.92) 
We recognize this as the log of a gamma distribution, and so identifying the coeffi- 
cients of a and ln a we obtain 


q (a) = Gam(alay, by) (10.93) 
where 
M 
an = aot 5 (10.94) 
1 
by = bo + 5E[w" J. (10.95) 


Similarly, we can find the variational re-estimation equation for the posterior 
distribution over w. Again, using the general result (10.9), and keeping only those 
terms that have a functional dependence on w, we have 


Ing*(w) = lnp(t|w) + Ea [In p(wla)] + const (10.96) 
ae ye —t,}? = 2 E[a]w* w + const (10.97) 
22W Pno ii a i 
1 
= aw (Eja]I + 6T) w+ Bw @"t+ const. (10.98) 


Because this is a quadratic form, the distribution q* (w) is Gaussian, and so we can 
complete the square in the usual way to identify the mean and covariance, giving 


q*(w) = N(w|my,Sw) (10.99) 
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where 


BSnye't (10.100) 
Sy = (Elali +688) `. (10.101) 


my 


Note the close similarity to the posterior distribution (3.52) obtained when a was 
treated as a fixed parameter. The difference is that here a is replaced by its expecta- 
tion Eja] under the variational distribution. Indeed, we have chosen to use the same 
notation for the covariance matrix Sy in both cases. 

Using the standard results (B.27), (B.38), and (B.39), we can obtain the required 
moments as follows 


Ela] = an/bn (10.102) 
E[ww'] = mymj)+ Sw. (10.103) 


The evaluation of the variational posterior distribution begins by initializing the pa- 
rameters of one of the distributions q(w) or q(a), and then alternately re-estimates 
these factors in turn until a suitable convergence criterion is satisfied (usually speci- 
fied in terms of the lower bound to be discussed shortly). 

It is instructive to relate the variational solution to that found using the evidence 
framework in Section 3.5. To do this consider the case ag = bọ = 0, corresponding 
to the limit of an infinitely broad prior over a. The mean of the variational posterior 
distribution g(a) is then given by 


, aN M/2 M 
Ela} = = = : 10.104 
lo] by Efjw?Tw]/2 mmy + Tr(Sy) \ 


Comparison with (9.63) shows that in the case of this particularly simple model, 
the variational approach gives precisely the same expression as that obtained by 
maximizing the evidence function using EM except that the point estimate for a 
is replaced by its expected value. Because the distribution g(w) depends on g(a) 
only through the expectation E/a], we see that the two approaches will give identical 
results for the case of an infinitely broad prior. 


10.3.2 Predictive distribution 


The predictive distribution over t, given a new input x, is easily evaluated for 
this model using the Gaussian variational posterior for the parameters 


px t) = / (tix, w)p(w|t) dw 


I 
Sa 

= 
x 
= 
= 
= 

a 

z 


- f NEW (x), elini 
EE (10.105) 
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where we have evaluated the integral by making use of the result (2.115) for the 
linear-Gaussian model. Here the input-dependent variance is given by 


o’ (x) = ; + 6(x)*Swd(x). (10.106) 


Note that this takes the same form as the result (3.59) obtained with fixed a except 
that now the expected value E[a] appears in the definition of Sy. 


10.3.3 Lower bound 
Another quantity of importance is the lower bound £ defined by 
L(q) = Ellnp(w,a,t)] — E[lng(w, a)] 


= Ey[np(tlw)] + Ew,alinp(wla)] + Ea{in p(o) 
—E, [In q(w)]w — Ellnq(a)]. (10.107) 


Evaluation of the various terms is straightforward, making use of results obtained in 
previous chapters, and gives 


N 
E[np(t|w)w = >In (=) ort + 8m 2" t 
on [f"S(mymj + Sy)] (10.108) 
Eflin p(wla)|w,a = — in(2n) + H Wlan) —Inby) 
E [mymy + Tr(Sy)| (10.109) 
E[In p(@)Ja = ao Inbo + (ao — 1) [W(an) — Indy] 
-bo — nT (an) (10.110) 
N 
—E[Ing(w)|w = smn [Sn] + x [1 + In(27)} (10.111) 
—E[Ing(a)ja = mT (an) — (an —1)(an) —Inby + ay. (10.112) 


Figure 10.9 shows a plot of the lower bound £(q) versus the degree of a polynomial 
model for a synthetic data set generated from a degree three polynomial. Here the 
prior parameters have been set to a9 = bp = 0, corresponding to the noninformative 
prior p(a) x 1/a, which is uniform over Ina as discussed in Section 2.3.6. As 
we saw in Section 10.1, the quantity £ represents lower bound on the log marginal 
likelihood p(t|M) for the model. If we assign equal prior probabilities p( M) to the 
different values of M, then we can interpret £ as an approximation to the poste- 
rior model probability p(/|t). Thus the variational framework assigns the highest 
probability to the model with M = 3. This should be contrasted with the maximum 
likelihood result, which assigns ever smaller residual error to models of increasing 
complexity until the residual error is driven to zero, causing maximum likelihood to 
favour severely over-fitted models. 
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Figure 10.9 Plot of the lower bound £ ver- 


10.4. 


sus the order M of the polyno- 
mial, for a polynomial model, in 
which a set of 10 data points is 
generated from a polynomial with 
M = 3 sampled over the inter- 
val (—5, 5) with additive Gaussian 
noise of variance 0.09. The value 
of the bound gives the log prob- 
ability of the model, and we see 
that the value of the bound peaks 
at M = 3, corresponding to the 
true model from which the data 
set was generated. 


Exponential Family Distributions 


In Chapter 2, we discussed the important role played by the exponential family of 
distributions and their conjugate priors. For many of the models discussed in this 
book, the complete-data likelihood is drawn from the exponential family. However, 
in general this will not be the case for the marginal likelihood function for the ob- 
served data. For example, in a mixture of Gaussians, the joint distribution of obser- 
vations Xn and corresponding hidden variables z,, is a member of the exponential 
family, whereas the marginal distribution of x,, is a mixture of Gaussians and hence 
is not. 

Up to now we have grouped the variables in the model into observed variables 
and hidden variables. We now make a further distinction between latent variables, 
denoted Z, and parameters, denoted 0, where parameters are intensive (fixed in num- 
ber independent of the size of the data set), whereas latent variables are extensive 
(scale in number with the size of the data set). For example, in a Gaussian mixture 
model, the indicator variables Zķn (which specify which component k is responsible 
for generating data point xn) represent the latent variables, whereas the means Hy, 
precisions A;, and mixing proportions 7; represent the parameters. 

Consider the case of independent identically distributed data. We denote the 
data values by X = {x,,}, where n = 1,...N, with corresponding latent variables 
Z = {z,,}. Now suppose that the joint distribution of observed and latent variables 
is a member of the a family, parameterized by natural parameters 77 so that 


p(X, Zin) = Ig (Xn, Zn)g(n) exp {nT u(Xn,Zn)} - (10.113) 


We shall also use a conjugate prior for 7, which can be written as 


p(n|vo, Vo) = f (Yo, Xo) 9(m)” exp {Von Xo} - (10.114) 


Recall that the conjugate prior distribution can be interpreted as a prior number vo 
of observations all having the value y, for the u vector. Now consider a variational 


Section 10.2.5 


10.4. Exponential Family Distributions 491 


distribution that factorizes between the latent variables and the parameters, so that 
q(Z,n) = q(Z)q(n). Using the general result (10.9), we can solve for the two 
factors as follows 


In q*(Z) 


E, (In p(X, Z|n)] + const 


N 
5 {In A(xn, Zn) + Efn ]u(xn, Zn) } + const. (10.115) 
n=1 


Thus we see that this decomposes into a sum of independent terms, one for each 
value of n, and hence the solution for q*(Z) will factorize over n so that q*(Z) = 
J [„ g“ (Zn). This is an example of an induced factorization. Taking the exponential 
of both sides, we have 


g (Zn) = h(Xn, Zn)g (Ely) exp { Eln ]u(xn, Zn) } (10.116) 


where the normalization coefficient has been re-instated by comparison with the 
standard form for the exponential family. 
Similarly, for the variational distribution over the parameters, we have 


In q*(n) = In p(n|vo, Xo) + Ez[ln p(X, Z|n)] + const (10.117) 


N 
= wlng(n) +n xo+ 5 {In g(m) + 9" Ez,,[U(Xn,Zn)]} + const. (10.118) 
n=1 


Again, taking the exponential of both sides, and re-instating the normalization coef- 
ficient by inspection, we have 


(n) = fun, Xn)g(n)”™ exp {7° xn } (10.119) 
where we have defined 


vn = Mw+N (10.120) 


N 
Xn = Xot X Es, [u(Kn,2n)]- (10.121) 
n=1 


Note that the solutions for q*(z,,) and q* (n) are coupled, and so we solve them iter- 
atively in a two-stage procedure. In the variational E step, we evaluate the expected 
sufficient statistics E[u(x,,,Z,)| using the current posterior distribution q(Zn) over 
the latent variables and use this to compute a revised posterior distribution q(7) over 
the parameters. Then in the subsequent variational M step, we use this revised pa- 
rameter posterior distribution to find the expected natural parameters E[n*], which 
gives rise to a revised variational distribution over the latent variables. 


10.4.1 Variational message passing 


We have illustrated the application of variational methods by considering a spe- 
cific model, the Bayesian mixture of Gaussians, in some detail. This model can be 
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described by the directed graph shown in Figure 10.5. Here we consider more gen- 
erally the use of variational methods for models described by directed graphs and 
derive a number of widely applicable results. 

The joint distribution corresponding to a directed graph can be written using the 
decomposition 


p(x) = | [ p(xilpa,) (10.122) 


where x; denotes the variable(s) associated with node 7, and pa; denotes the parent 
set corresponding to node 7. Note that x; may be a latent variable or it may belong 
to the set of observed variables. Now consider a variational approximation in which 
the distribution g(x) is assumed to factorize with respect to the x; so that 


a(x) = | [ aii). (10.123) 


Note that for observed nodes, there is no factor q(x;) in the variational distribution. 
We now substitute (10.122) into our general result (10.9) to give 


Ing} (x;) = Eizy X a) + const. (10.124) 


Any terms on the right-hand side that do not depend on x; can be absorbed into 
the additive constant. In fact, the only terms that do depend on x; are the con- 
ditional distribution for x; given by p(x;|pa,) together with any other conditional 
distributions that have x; in the conditioning set. By definition, these conditional 
distributions correspond to the children of node j, and they therefore also depend on 
the co-parents of the child nodes, i.e., the other parents of the child nodes besides 
node x; itself. We see that the set of all nodes on which q*(x;) depends corresponds 
to the Markov blanket of node x,, as illustrated in Figure 8.26. Thus the update 
of the factors in the variational posterior distribution represents a local calculation 
on the graph. This makes possible the construction of general purpose software for 
variational inference in which the form of the model does not need to be specified in 
advance (Bishop et al., 2003). 

If we now specialize to the case of a model in which all of the conditional dis- 
tributions have a conjugate-exponential structure, then the variational update proce- 
dure can be cast in terms of a local message passing algorithm (Winn and Bishop, 
2005). In particular, the distribution associated with a particular node can be updated 
once that node has received messages from all of its parents and all of its children. 
This in turn requires that the children have already received messages from their co- 
parents. The evaluation of the lower bound can also be simplified because many of 
the required quantities are already evaluated as part of the message passing scheme. 
This distributed message passing formulation has good scaling properties and is well 
suited to large networks. 
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10.5. Local Variational Methods 


Section 1.6.1 


The variational framework discussed in Sections 10.1 and 10.2 can be considered a 
‘global’ method in the sense that it directly seeks an approximation to the full poste- 
rior distribution over all random variables. An alternative ‘local’ approach involves 
finding bounds on functions over individual variables or groups of variables within 
a model. For instance, we might seek a bound on a conditional distribution p(y|z), 
which is itself just one factor in a much larger probabilistic model specified by a 
directed graph. The purpose of introducing the bound of course is to simplify the 
resulting distribution. This local approximation can be applied to multiple variables 
in turn until a tractable approximation is obtained, and in Section 10.6.1 we shall 
give a practical example of this approach in the context of logistic regression. Here 
we focus on developing the bounds themselves. 

We have already seen in our discussion of the Kullback-Leibler divergence that 
the convexity of the logarithm function played a key role in developing the lower 
bound in the global variational approach. We have defined a (strictly) convex func- 
tion as one for which every chord lies above the function. Convexity also plays a 
central role in the local variational framework. Note that our discussion will ap- 
ply equally to concave functions with ‘min’ and ‘max’ interchanged and with lower 
bounds replaced by upper bounds. 

Let us begin by considering a simple example, namely the function f(x) = 
exp(—), which is a convex function of x, and which is shown in the left-hand plot 
of Figure 10.10. Our goal is to approximate f(x) by a simpler function, in particular 
a linear function of x. From Figure 10.10, we see that this linear function will be a 
lower bound on f(x) if it corresponds to a tangent. We can obtain the tangent line 
y(x) at a specific value of x, say x = €, by making a first order Taylor expansion 


y(x) = FE) + FE- E) (10.125) 


so that y(x) < f(x) with equality when x = €. For our example function f(x) = 


Figure 10.10 In the left-hand fig- 
ure the red curve shows the function 
exp(—x), and the blue line shows 
the tangent at xv = € defined by 
(10.125) with € = 1. This line has 
slope \ = f’(€) = —exp(—é). Note 
that any other tangent line, for ex- 
ample the ones shown in green, will 
have a smaller value of y at x = 
€. The right-hand figure shows the 
corresponding plot of the function 
AE — g(A), where g(A) is given by 
(10.131), versus A for € = 1, in 
which the maximum corresponds to 
à = —exp(—€) = —1/e. 


1 0.4 
AE — g(r) 

0.5 0.2 

0 0 
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In the left-hand plot the red curve shows a convex function f(x), and the blue line represents the 


linear function Az, which is a lower bound on f(x) because f(x) > Az for all x. For the given value of slope à the 
contact point of the tangent line having the same slope is found by minimizing with respect to x the discrepancy 
(shown by the green dashed lines) given by f(x) — Ax. This defines the dual function g(A), which corresponds 
to the (negative of the) intercept of the tangent line having slope à. 


exp(—2), we therefore obtain the tangent line in the form 


y(x) = exp(—€) — exp(—&)(a — £) (10.126) 
which is a linear function parameterized by €. For consistency with subsequent 
discussion, let us define A = — exp(—&) so that 

ylz, A) = Aw — A+ Aln(-A). (10.127) 


Different values of A correspond to different tangent lines, and because all such lines 
are lower bounds on the function, we have f(x) > y(x, A). Thus we can write the 
function in the form 


f(ay= max {Ax — àA +àln(—A)}. (10.128) 


We have succeeded in approximating the convex function f(x) by a simpler, lin- 
ear function y(x, \). The price we have paid is that we have introduced a variational 
parameter À, and to obtain the tightest bound we must optimize with respect to À. 

We can formulate this approach more generally using the framework of convex 
duality (Rockafellar, 1972; Jordan et al., 1999). Consider the illustration of a convex 
function f(x) shown in the left-hand plot in Figure 10.11. In this example, the 
function Ax is a lower bound on f(x) but it is not the best lower bound that can 
be achieved by a linear function having slope A, because the tightest bound is given 
by the tangent line. Let us write the equation of the tangent line, having slope A as 
Ax — g(A) where the (negative) intercept g(A) clearly depends on the slope A of the 
tangent. To determine the intercept, we note that the line must be moved vertically by 
an amount equal to the smallest vertical distance between the line and the function, 
as shown in Figure 10.11. Thus 


gA) = - min {f (x) — Ax} 
max {rAx — f(x)}. (10.129) 
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Now, instead of fixing À and varying x, we can consider a particular x and then 
adjust À until the tangent plane is tangent at that particular x. Because the y value 
of the tangent line at a particular x is maximized when that value coincides with its 
contact point, we have 

fa)= max {Ax — g(à)}. (10.130) 


We see that the functions f(x) and g(A) play a dual role, and are related through 
(10.129) and (10.130). 

Let us apply these duality relations to our simple example f(x) = exp(—z). 
From (10.129) we see that the maximizing value of x is given by € = — In(—A), and 
back-substituting we obtain the conjugate function g(A) in the form 


g(A) = à — Aln(—A) (10.131) 


as obtained previously. The function AE — g(A) is shown, for € = 1 in the right-hand 
plot in Figure 10.10. As a check, we can substitute (10.131) into (10.130), which 
gives the maximizing value of \ = — exp(—2), and back-substituting then recovers 
the original function f(x) = exp(—2). 

For concave functions, we can follow a similar argument to obtain upper bounds, 
in which max’ is replaced with ‘min’, so that 


ia) = min {Ax — g(A)} (10.132) 
min {Ax — f(x)}. (10.133) 


Q 

— 
> 

ai 
II 


If the function of interest is not convex (or concave), then we cannot directly 
apply the method above to obtain a bound. However, we can first seek invertible 
transformations either of the function or of its argument which change it into a con- 
vex form. We then calculate the conjugate function and then transform back to the 
original variables. 

An important example, which arises frequently in pattern recognition, is the 
logistic sigmoid function defined by 


1 


= — 10.134 
Ieper? nen 


a(x) 
As it stands this function is neither convex nor concave. However, if we take the 
logarithm we obtain a function which is concave, as is easily verified by finding the 
second derivative. From (10.133) the corresponding conjugate function then takes 
the form 


g(A) = min {Ax — f(z)} = —àln à — (1 — A) In(1 — A) (10.135) 


which we recognize as the binary entropy function for a variable whose probability 
of having the value 1 is A. Using (10.132), we then obtain an upper bound on the log 
sigmoid 

Ino(x) < Ax — g(A) (10.136) 


496 10. APPROXIMATE INFERENCE 


0.5 F 


l 
l 
l 
l 
l 
l 
l 
l 
, l 
~6 0 6 ~6 =f 0 é 6 


Figure 10.12 The left-hand plot shows the logistic sigmoid function o(a) defined by (10.134) in red, together 
with two examples of the exponential upper bound (10.137) shown in blue. The right-hand plot shows the logistic 
sigmoid again in red together with the Gaussian lower bound (10.144) shown in blue. Here the parameter 
€ = 2.5, and the bound is exact at x = € and x = —€, denoted by the dashed green lines. 


and taking the exponential, we obtain an upper bound on the logistic sigmoid itself 
of the form 
a(x) < exp(Az — g(A)) (10.137) 


which is plotted for two values of À on the left-hand plot in Figure 10.12. 

We can also obtain a lower bound on the sigmoid having the functional form of 
a Gaussian. To do this, we follow Jaakkola and Jordan (2000) and make transforma- 
tions both of the input variable and of the function itself. First we take the log of the 
logistic function and then decompose it so that 


Ino(z) = —-In(l+e°7)=-In fe erer a e772) 
= g/2— ln(e”? + e7”). (10.138) 
We now note that the function f(x) = —In(e”/? + e~*/?) is a convex function of 
Exercise 10.31 the variable x”, as can again be verified by finding the second derivative. This leads 
to a lower bound on f(x), which is a linear function of x? whose conjugate function 
is given by 
g(A) = max {dx -f (va?) (10.139) 
The stationarity condition leads to 
dx d 1 x 
o=- sie) =r+ — tanh (Z). 10.140 
dz? dx (2) i ge ONR ( ) 


If we denote this value of x, corresponding to the contact point of the tangent line 
for this particular value of A, by £, then we have 


ME) = zz tann ($) = : K |. (10.141) 
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Instead of thinking of A as the variational parameter, we can let € play this role as 
this leads to simpler expressions for the conjugate function, which is then given by 


9A) = AEE? — F(E) = AOE? + In(e? + e£/?). (10.142) 
Hence the bound on f(x) can be written as 

f(a) > Aa? — g(A) = Aw? — AE? — In(e&/? + e78?) (10.143) 
The bound on the sigmoid then becomes 


a(x) > o(€) exp { (x — £)/2 — AE) (2? — &) } (10.144) 


where (€) is defined by (10.141). This bound is illustrated in the right-hand plot of 
Figure 10.12. We see that the bound has the form of the exponential of a quadratic 
function of x, which will prove useful when we seek Gaussian representations of 
posterior distributions defined through logistic sigmoid functions. 

The logistic sigmoid arises frequently in probabilistic models over binary vari- 
ables because it is the function that transforms a log odds ratio into a posterior prob- 
ability. The corresponding transformation for a multiclass distribution is given by 
the softmax function. Unfortunately, the lower bound derived here for the logistic 
sigmoid does not directly extend to the softmax. Gibbs (1997) proposes a method 
for constructing a Gaussian distribution that is conjectured to be a bound (although 
no rigorous proof is given), which may be used to apply local variational methods to 
multiclass problems. 

We shall see an example of the use of local variational bounds in Sections 10.6.1. 
For the moment, however, it is instructive to consider in general terms how these 
bounds can be used. Suppose we wish to evaluate an integral of the form 


I= frora da (10.145) 


where o(a) is the logistic sigmoid, and p(a) is a Gaussian probability density. Such 
integrals arise in Bayesian models when, for instance, we wish to evaluate the pre- 
dictive distribution, in which case p(a) represents a posterior parameter distribution. 
Because the integral is intractable, we employ the variational bound (10.144), which 
we write in the form o(a) > f(a,&) where € is a variational parameter. The inte- 
gral now becomes the product of two exponential-quadratic functions and so can be 
integrated analytically to give a bound on T 


I> I f(a,€)p(a) da = F(8). (10.146) 


We now have the freedom to choose the variational parameter €, which we do by 
finding the value ¿* that maximizes the function F(£). The resulting value F'(é*) 
represents the tightest bound within this family of bounds and can be used as an 
approximation to I. This optimized bound, however, will in general not be exact. 
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Although the bound o(a) > f(a, €) on the logistic sigmoid can be optimized exactly, 
the required choice for € depends on the value of a, so that the bound is exact for one 
value of a only. Because the quantity F (£) is obtained by integrating over all values 
of a, the value of €* represents a compromise, weighted by the distribution p(a). 


Variational Logistic Regression 


We now illustrate the use of local variational methods by returning to the Bayesian 
logistic regression model studied in Section 4.5. There we focussed on the use of 
the Laplace approximation, while here we consider a variational treatment based on 
the approach of Jaakkola and Jordan (2000). Like the Laplace method, this also 
leads to a Gaussian approximation to the posterior distribution. However, the greater 
flexibility of the variational approximation leads to improved accuracy compared 
to the Laplace method. Furthermore (unlike the Laplace method), the variational 
approach is optimizing a well defined objective function given by a rigourous bound 
on the model evidence. Logistic regression has also been treated by Dybowski and 
Roberts (2005) from a Bayesian perspective using Monte Carlo sampling techniques. 


10.6.1 Variational posterior distribution 


Here we shall make use of a variational approximation based on the local bounds 
introduced in Section 10.5. This allows the likelihood function for logistic regres- 
sion, which is governed by the logistic sigmoid, to be approximated by the expo- 
nential of a quadratic form. It is therefore again convenient to choose a conjugate 
Gaussian prior of the form (4.140). For the moment, we shall treat the hyperparam- 
eters mo and So as fixed constants. In Section 10.6.3, we shall demonstrate how the 
variational formalism can be extended to the case where there are unknown hyper- 
parameters whose values are to be inferred from the data. 

In the variational framework, we seek to maximize a lower bound on the marginal 
likelihood. For the Bayesian logistic regression model, the marginal likelihood takes 
the form 


N 
p(t) = 1 p(t|w)p(w) dw = f TI Him) p(w) dw. (10.147) 


We first note that the conditional distribution for t can be written as 


ptlw) = o(a) {1- o(a)} 
t 1—t 
_ 1 i 1 
(5) ( =) 
= at e* — vat — 
UEF o(—a) (10.148) 


where a = wọ. In order to obtain a lower bound on p(t), we make use of the 
variational lower bound on the logistic sigmoid function given by (10.144), which 
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we reproduce here for convenience 


a(z) > o(€)exp {(z — £)/2 — AE (2? - &)} (10.149) 
where ; i 
AE) = 2€ ove) = l ' (10.150) 


We can therefore write 


p(tlw) = e*a(—a) > e%o(£) exp {—(a + €)/2 — AE) (a? — €&)}. (10.151) 


Note that because this bound is applied to each of the terms in the likelihood function 
separately, there is a variational parameter €,, corresponding to each training set 
observation (@,,, tn). Using a = w' @, and multiplying by the prior distribution, we 
obtain the following bound on the joint distribution of t and w 


p(t, w) = p(t|w)p(w) > h(w, €)p(w) (10.152) 


where € denotes the set {€,, } of variational parameters, and 


N 
h(w,€) = [J] o&n)exp {wontn — (wen + &n)/2 


n=1 


— (En) ([w" bn]? — &)}. (10.153) 


Evaluation of the exact posterior distribution would require normalization of the left- 
hand side of this inequality. Because this is intractable, we work instead with the 
right-hand side. Note that the function on the right-hand side cannot be interpreted 
as a probability density because it is not normalized. Once it is normalized to give a 
variational posterior distribution q(w), however, it no longer represents a bound. 

Because the logarithm function is monotonically increasing, the inequality A > 
B implies ln A > In B. This gives a lower bound on the log of the joint distribution 
of t and w of the form 


N 


In {p(t]w)p(w)} > In p(w) + oe {Ino(En) + w' dptn 


— CERTE E Pnl — ER). (10.154) 


Substituting for the prior p(w), the right-hand side of this inequality becomes, as a 
function of w 


1 
= — mo) "S3 (w = mo) 


N 
+Y {win (tn — 1/2) — Alén)wT (Øn On)w} + const. (10.155) 
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This is a quadratic function of w, and so we can obtain the corresponding variational 
approximation to the posterior distribution by identifying the linear and quadratic 
terms in w, giving a Gaussian variational posterior of the form 


q(w) = N(w|my,Sy) (10.156) 
where 
N 
my = Sn [sim ene F a = 12) (10.157) 
n=1 
N 
Sy = B PIN > AMEn)ondr- (10.158) 
n=l 


As with the Laplace framework, we have again obtained a Gaussian approximation 
to the posterior distribution. However, the additional flexibility provided by the vari- 
ational parameters {€,,} leads to improved accuracy in the approximation (Jaakkola 
and Jordan, 2000). 

Here we have considered a batch learning context in which all of the training 
data is available at once. However, Bayesian methods are intrinsically well suited 
to sequential learning in which the data points are processed one at a time and then 
discarded. The formulation of this variational approach for the sequential case is 
straightforward. 

Note that the bound given by (10.149) applies only to the two-class problem and 
so this approach does not directly generalize to classification problems with K > 2 
classes. An alternative bound for the multiclass case has been explored by Gibbs 
(1997). 


10.6.2 Optimizing the variational parameters 


We now have a normalized Gaussian approximation to the posterior distribution, 
which we shall use shortly to evaluate the predictive distribution for new data points. 
First, however, we need to determine the variational parameters {€,, } by maximizing 
the lower bound on the marginal likelihood. 

To do this, we substitute the inequality (10.152) back into the marginal likeli- 
hood to give 


In p(t) = in f v(thw)p(w) dw > m f his, £pl) dw = £L (€). (10.159) 


As with the optimization of the hyperparameter œ in the linear regression model of 
Section 3.5, there are two approaches to determining the €,,. In the first approach, we 
recognize that the function £(€) is defined by an integration over w and so we can 
view w as a latent variable and invoke the EM algorithm. In the second approach, 
we integrate over w analytically and then perform a direct maximization over €. Let 
us begin by considering the EM approach. 

The EM algorithm starts by choosing some initial values for the parameters 
{én}, which we denote collectively by €°'". In the E step of the EM algorithm, 
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we then use these parameter values to find the posterior distribution over w, which 
is given by (10.156). In the M step, we then maximize the expected complete-data 
log likelihood which is given by 


Q(Eé, E£”) = E [In h(w, €)p(w)] (10.160) 


where the expectation is taken with respect to the posterior distribution g(w) evalu- 


ated using °°. Noting that p(w) does not depend on £, and substituting for h(w, £) 
we obtain 
N 
QE, E") = X {Ino(&n) — £n/2 — MEn) (Gn Elww"]d, — £) } + const 
n=1 


(10.161) 
where ‘const’ denotes terms that are independent of €. We now set the derivative with 
respect to €,, equal to zero. A few lines of algebra, making use of the definitions of 
o(€) and A(£), then gives 


0 = (En)(¢,E[ww"]d,, — £2). (10.162) 


We now note that A’ (£) is a monotonic function of € for € > 0, and that we can 
restrict attention to nonnegative values of € without loss of generality due to the 
symmetry of the bound around € = 0. Thus X’(€) # 0, and hence we obtain the 
following re-estimation equations 


(grew)? — on ElwwT]o, = p: (Sn Pa mymy) Pn (10.163) 


where we have used (10.156). 

Let us summarize the EM algorithm for finding the variational posterior distri- 
bution. We first initialize the variational parameters €°'". In the E step, we evaluate 
the posterior distribution over w given by (10.156), in which the mean and covari- 
ance are defined by (10.157) and (10.158). In the M step, we then use this variational 
posterior to compute a new value for € given by (10.163). The E and M steps are 
repeated until a suitable convergence criterion is satisfied, which in practice typically 
requires only a few iterations. 

An alternative approach to obtaining re-estimation equations for € is to note 
that in the integral over w in the definition (10.159) of the lower bound £(&), the 
integrand has a Gaussian-like form and so the integral can be evaluated analytically. 
Having evaluated the integral, we can then differentiate with respect to n. It turns 
out that this gives rise to exactly the same re-estimation equations as does the EM 
approach given by (10.163). 

As we have emphasized already, in the application of variational methods it is 
useful to be able to evaluate the lower bound £ (€) given by (10.159). The integration 
over w can be performed analytically by noting that p(w) is Gaussian and h(w, €) 
is the exponential of a quadratic function of w. Thus, by completing the square 
and making use of the standard result for the normalization coefficient of a Gaussian 
distribution, we can obtain a closed form solution which takes the form 
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Figure 10.13 
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Illustration of the Bayesian approach to logistic regression for a simple linearly separable data 


set. The plot on the left shows the predictive distribution obtained using variational inference. We see that 
the decision boundary lies roughly mid way between the clusters of data points, and that the contours of the 
predictive distribution splay out away from the data reflecting the greater uncertainty in the classification of such 
regions. The plot on the right shows the decision boundaries corresponding to five samples of the parameter 
vector w drawn from the posterior distribution p(w|t). 
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This variational framework can also be applied to situations in which the data 
is arriving sequentially (Jaakkola and Jordan, 2000). In this case we maintain a 
Gaussian posterior distribution over w, which is initialized using the prior p(w). As 
each data point arrives, the posterior is updated by making use of the bound (10.151) 
and then normalized to give an updated posterior distribution. 

The predictive distribution is obtained by marginalizing over the posterior dis- 
tribution, and takes the same form as for the Laplace approximation discussed in 
Section 4.5.2. Figure 10.13 shows the variational predictive distributions for a syn- 
thetic data set. This example provides interesting insights into the concept of ‘large 
margin’, which was discussed in Section 7.1 and which has qualitatively similar be- 
haviour to the Bayesian solution. 


10.6.3 Inference of hyperparameters 


So far, we have treated the hyperparameter a in the prior distribution as a known 
constant. We now extend the Bayesian logistic regression model to allow the value of 
this parameter to be inferred from the data set. This can be achieved by combining 
the global and local variational approximations into a single framework, so as to 
maintain a lower bound on the marginal likelihood at each stage. Such a combined 
approach was adopted by Bishop and Svensén (2003) in the context of a Bayesian 
treatment of the hierarchical mixture of experts model. 
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Specifically, we consider once again a simple isotropic Gaussian prior distribu- 
tion of the form 
p(wla) = N(w|0, 07"). (10.165) 


Our analysis is readily extended to more general Gaussian priors, for instance if we 
wish to associate a different hyperparameter with different subsets of the parame- 
ters wj. As usual, we consider a conjugate hyperprior over œ given by a gamma 
distribution 

pla) = Gam(alao, bo) (10.166) 


governed by the constants ao and bo. 
The marginal likelihood for this model now takes the form 


= f p(w, a,t) dwda (10.167) 


where the joint distribution is given by 
p(w, a, t) = p(t|w)p(wla)p(a). (10.168) 


We are now faced with an analytically intractable integration over w and a, which 
we Shall tackle by using both the local and global variational approaches in the same 
model 

To begin with, we introduce a variational distribution g(w, a), and then apply 
the decomposition (10.2), which in this instance takes the form 


In p(t) = L(g) + KL(q|lp) (10.169) 


where the lower bound £(q) and the Kullback-Leibler divergence KL(q||p) are de- 
fined by 


q(w 


- ffa (w,a) ng pate | dwda. (10.171) 


At this point, the lower bound £(q) is still intractable due to the form of the 
likelihood factor p(t|w). We therefore apply the local variational bound to each of 
the logistic sigmoid factors as before. This allows us to use the inequality (10.152) 
and place a lower bound on £(q), which will therefore also be a lower bound on the 
log marginal likelihood 


Inp(t) 2 > L(q, €) 
7 (wa yin MEP ate dwda. (10.172) 


q(w, a) 


Lla) = | fa (w, a mf? BAW meai dw da (10.170) 


KL(ql|p) 


Next we assume that the variational distribution factorizes between parameters and 
hyperparameters so that 


q(w,a) = q(w)q(a). (10.173) 
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With this factorization we can appeal to the general result (10.9) to find expressions 
for the optimal factors. Consider first the distribution g(w). Discarding terms that 
are independent of w, we have 


Ing(w) = Eq [In {h(w,€)p(wla)p(a)}] + const 
= Inh(w,&) + Ea [In p(wla)] + const. 


We now substitute for In h(w, €) using (10.153), and for In p(w|a) using (10.165), 


giving 
, N 
Ing(w) = — a w'w4 5 {(tn —1/2)w'd,, — MEn)wibnP, Ww} + const. 
n=1 


We see that this is a quadratic function of w and so the solution for g(w) will be 
Gaussian. Completing the square in the usual way, we obtain 


gw) = N(w|py, En) (10.174) 
where we have defined 
N 
Sy in = > (ne —1/2)0, (10.175) 
n=1 
N 
Sy = Eloll +2) AEn) pno. (10.176) 
n=1 


Similarly, the optimal solution for the factor g(a) is obtained from 


In g(a) = Ey [Inp(w|a)] + In p(a) + const. 


Substituting for In p(w|a) using (10.165), and for In p(a) using (10.166), we obtain 


M 
Ing(a) = = Ina — 5 E [wi w] + (ao — 1) na — boa + const. 


We recognize this as the log of a gamma distribution, and so we obtain 


1 bo ,ap—-1,—b 
qla) = Gam(alay, by) = a ArT ne 78" (10.177) 
( ) ( | N N) T(ao) (0) 
where 
M 
an = ao + > (10.178) 
1 
by = bo + Ew[w'w]. (10.179) 
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We also need to optimize the variational parameters €,,, and this is also done by 


maximizing the lower bound Liq, €). Omitting terms that are independent of €, and 
integrating over a, we have 


L(q,€) = aco) mn(w, £) dw + const (10.180) 


Note that this has precisely the same form as (10.159), and so we can again appeal 
to our earlier result (10.163), which can be obtained by direct optimization of the 
marginal likelihood function, leading to re-estimation equations of the form 


We have obtained re-estimation equations for the three quantities g(w), q(a), 
and &, and so after making suitable initializations, we can cycle through these quan- 
tities, updating each in turn. The required moments are given by 


Ele) = —* (10.182) 
bn 
E[w'w] = Un+pyen- (10.183) 


Expectation Propagation 


We conclude this chapter by discussing an alternative form of deterministic approx- 
imate inference, known as expectation propagation or EP (Minka, 2001a; Minka, 
2001b). As with the variational Bayes methods discussed so far, this too is based 
on the minimization of a Kullback-Leibler divergence but now of the reverse form, 
which gives the approximation rather different properties. 

Consider for a moment the problem of minimizing KL(p||q) with respect to q(z) 
when p(z) is a fixed distribution and q(z) is a member of the exponential family and 
so, from (2.194), can be written in the form 


q(z) = h(z)g(m) exp {n™u(z)}. (10.184) 
As a function of 7, the Kullback-Leibler divergence then becomes 


KL(p||¢) = — In g(n) — 7 Ep@y[u(z)] + const (10.185) 


where the constant terms are independent of the natural parameters 7. We can mini- 
mize KL(p||q) within this family of distributions by setting the gradient with respect 
to 7 to zero, giving 


-V Ing(n) = Ep) lu(z)]. (10.186) 


However, we have already seen in (2.226) that the negative gradient of ln g(7) is 
given by the expectation of u(z) under the distribution q(z). Equating these two 
results, we obtain 


Eq(z)[u(z)] = Epc) [u(z)). (10.187) 


506 


10. APPROXIMATE INFERENCE 


We see that the optimum solution simply corresponds to matching the expected suf- 
ficient statistics. So, for instance, if g(z) is a Gaussian N (z|, X) then we minimize 
the Kullback-Leibler divergence by setting the mean ys of g(z) equal to the mean of 
the distribution p(z) and the covariance & equal to the covariance of p(z). This is 
sometimes called moment matching. An example of this was seen in Figure 10.3(a). 
Now let us exploit this result to obtain a practical algorithm for approximate 
inference. For many probabilistic models, the joint distribution of data D and hidden 
variables (including parameters) @ comprises a product of factors in the form 


p(D, 8) = ] [ £40). (10.188) 


This would arise, for example, in a model for independent, identically distributed 
data in which there is one factor f,,(@) = p(x,|@) for each data point x,,, along 
with a factor fo(@) = p(@) corresponding to the prior. More generally, it would also 
apply to any model defined by a directed probabilistic graph in which each factor is a 
conditional distribution corresponding to one of the nodes, or an undirected graph in 
which each factor is a clique potential. We are interested in evaluating the posterior 
distribution p(@|D) for the purpose of making predictions, as well as the model 
evidence p(D) for the purpose of model comparison. From (10.188) the posterior is 
given by 


p(O|D) = ap lL) (10.189) 


and the model evidence is given by 
p(D) = fil fi(0) da. (10.190) 


Here we are considering continuous variables, but the following discussion applies 
equally to discrete variables with integrals replaced by summations. We shall sup- 
pose that the marginalization over 0, along with the marginalizations with respect to 
the posterior distribution required to make predictions, are intractable so that some 
form of approximation is required. 

Expectation propagation is based on an approximation to the posterior distribu- 
tion which is also given by a product of factors 


q(0) = BICO (10.191) 


in which each factor f;(@) in the approximation corresponds to one of the factors 
fi(@) in the true posterior (10.189), and the factor 1/Z is the normalizing constant 
needed to ensure that the left-hand side of (10.191) integrates to unity. In order to 
obtain a practical algorithm, we need to constrain the factors f;(@) in some way, 
and in particular we shall assume that they come from the exponential family. The 
product of the factors will therefore also be from the exponential family and so can 
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be described by a finite set of sufficient statistics. For example, if each of the fi(0) 
is a Gaussian, then the overall approximation q(@) will also be Gaussian. 

Ideally we would like to determine the f;(@) by minimizing the Kullback-Leibler 
divergence between the true posterior and the approximation given by 


1 1 ~ 
L (p||¢) = KL (a I+) z Hi) (10.192) 


Note that this is the reverse form of KL divergence compared with that used in varia- 
tional inference. In general, this minimization will be intractable because the KL di- 
vergence involves averaging with respect to the true distribution. As a rough approx- 
imation, we could instead minimize the KL divergences between the corresponding 
pairs f;(@) and f;(@) of factors. This represents a much simpler problem to solve, 
and has the advantage that the algorithm is noniterative. However, because each fac- 
tor is individually approximated, the product of the factors could well give a poor 
approximation. 

Expectation propagation makes a much better approximation by optimizing each 
factor in turn in the context of all of the remaining factors. It starts by initializing 
the factors f;(@), and then cycles through the factors refining them one at a time. 
This is similar in spirit to the update of factors in the variational Bayes framework 
considered earlier. Suppose we wish to refine factor f;(@). We first remove this 


factor from the product to give |, Zi fi(0). Conceptually, we will now determine a 


revised form of the factor ee by ensuring that the product 


g (9) x f;(9) [ [F0 (10.193) 
ifj 
is as close as possible to 
0) | | f:(9) (10.194) 
ifj 


in which we keep fixed all of the factors f;(@) for i # j. This ensures that the 
approximation is most accurate in the regions of high posterior probability as defined 
by the remaining factors. We shall see an example of this effect when we apply EP 
to the ‘clutter problem’. To achieve this, we first remove the factor f;(@) from the 
current approximation to the posterior by defining the unnormalized distribution 


0 
\J(@) = (8) 
f(4) 
Note that we could instead find qV (0) from the product of factors i # j, although 


in practice division is usually easier. This is now combined with the factor f;(0) to 
give a distribution 


(10.195) 


> F(a (0) (10.196) 
Zj 
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Illustration of the expectation propagation approximation using a Gaussian distribution for the 


example considered earlier in Figures 4.14 and 10.1. The left-hand plot shows the original distribution (yellow) 
along with the Laplace (red), global variational (green), and EP (blue) approximations, and the right-hand plot 
shows the corresponding negative logarithms of the distributions. Note that the EP distribution is broader than 
that variational inference, as a consequence of the different form of KL divergence. 


where Z; is the normalization constant given by 
Z= [1% dé. (10.197) 


We now determine a revised factor fe (0) by minimizing the Kullback-Leibler diver- 


gence 
f;(0)a (8) 
KL (art 


j 


a (0)) . (10.198) 


This is easily solved because the approximating distribution g"°” (0) is from the ex- 
ponential family, and so we can appeal to the result (10.187), which tells us that the 
parameters of q™°¥ (0) are obtained by matching its expected sufficient statistics to 
the corresponding moments of (10.196). We shall assume that this is a tractable oper- 
ation. For example, if we choose q(@) to be a Gaussian distribution N (0| u, ©), then 
p is set equal to the mean of the (unnormalized) distribution f;(@)q\! (0), and © is 
set to its covariance. More generally, it is straightforward to obtain the required ex- 
pectations for any member of the exponential family, provided it can be normalized, 
because the expected statistics can be related to the derivatives of the normalization 
coefficient, as given by (2.226). The EP approximation is illustrated in Figure 10.14. 


From (10.193), we see that the revised factor f;(9) can be found by taking 
q™°™ (0) and dividing out the remaining factors so that 


gq (0) 
qv) (8) 


where we have used (10.195). The coefficient K is determined by multiplying both 


f;(0) =K (10.199) 
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sides of (10.199) by q\‘(@) and integrating to give 


K= f F;(0)q™ (0) d0 (10.200) 


where we have used the fact that q”°™ (0) is normalized. The value of K can therefore 
be found by matching zeroth-order moments 


[F000 | 0O). (10.201) 


Combining this with (10.197), we then see that K = Z; and so can be found by 
evaluating the integral in (10.197). 

In practice, several passes are made through the set of factors, revising each 
factor in turn. The posterior distribution p(@|D) is then approximated using (10.191), 
and the model evidence p(D) can be approximated by using (10.190) with the factors 


f;(@) replaced by their approximations f;(0). 
Expectation Propagation 


We are given a joint distribution over observed data D and stochastic variables 
0 in the form of a product of factors 


p(D, 6) = || fi) (10.202) 


and we wish to approximate the posterior distribution p(@|D) by a distribution 
of the form 


1 
a8) = z [40 (10.203) 
We also wish to approximate the model evidence p(D). 


1. Initialize all of the approximating factors fi(@). 


2. Initialize the posterior approximation by setting 


q(9) x | | F0). (10.204) 


3. Until convergence: 
(a) Choose a factor re (0) to refine. 
(b) Remove $; (0) from the posterior by division 


qi (0) = 2. (10.205) 
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(c) Evaluate the new posterior by setting the sufficient statistics (moments) 
of q™°™ (0) equal to those of q“ (0) f;(0), including evaluation of the 
normalization constant 


Zj = 1 q (0) f;(0) dd. (10.206) 
(d) Evaluate and store the new factor 
gq’ (0) 


f;(0) = 2i TO (10.207) 


4. Evaluate the approximation to the model evidence 


p(D) ~ f io dé. (10.208) 


A special case of EP, known as assumed density filtering (ADF) or moment 
matching (Maybeck, 1982; Lauritzen, 1992; Boyen and Koller, 1998; Opper and 
Winther, 1999), is obtained by initializing all of the approximating factors except 
the first to unity and then making one pass through the factors updating each of them 
once. Assumed density filtering can be appropriate for on-line learning in which data 
points are arriving in a sequence and we need to learn from each data point and then 
discard it before considering the next point. However, in a batch setting we have the 
opportunity to re-use the data points many times in order to achieve improved ac- 
curacy, and it is this idea that is exploited in expectation propagation. Furthermore, 
if we apply ADF to batch data, the results will have an undesirable dependence on 
the (arbitrary) order in which the data points are considered, which again EP can 
overcome. 

One disadvantage of expectation propagation is that there is no guarantee that 
the iterations will converge. However, for approximations q(0) in the exponential 
family, if the iterations do converge, the resulting solution will be a stationary point 
of a particular energy function (Minka, 2001a), although each iteration of EP does 
not necessarily decrease the value of this energy function. This is in contrast to 
variational Bayes, which iteratively maximizes a lower bound on the log marginal 
likelihood, in which each iteration is guaranteed not to decrease the bound. It is 
possible to optimize the EP cost function directly, in which case it is guaranteed 
to converge, although the resulting algorithms can be slower and more complex to 
implement. 

Another difference between variational Bayes and EP arises from the form of 
KL divergence that is minimized by the two algorithms, because the former mini- 
mizes KL(q||p) whereas the latter minimizes KL(p]||q). As we saw in Figure 10.3, 
for distributions p(@) which are multimodal, minimizing KL(p||q) can lead to poor 
approximations. In particular, if EP is applied to mixtures the results are not sen- 
sible because the approximation tries to capture all of the modes of the posterior 
distribution. Conversely, in logistic-type models, EP often out-performs both local 
variational methods and the Laplace approximation (Kuss and Rasmussen, 2006). 
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Illustration of the clutter problem 
for a data space dimensionality of 
D = 1. Training data points, de- 
noted by the crosses, are drawn 
from a mixture of two Gaussians 
with components shown in red 
and green. The goal is to infer the 
mean of the green Gaussian from 
the observed data. 


10.7.1 Example: The clutter problem 


Following Minka (2001b), we illustrate the EP algorithm using a simple exam- 
ple in which the goal is to infer the mean 0 of a multivariate Gaussian distribution 
over a variable x given a set of observations drawn from that distribution. To make 
the problem more interesting, the observations are embedded in background clutter, 
which itself is also Gaussian distributed, as illustrated in Figure 10.15. The distribu- 
tion of observed values x is therefore a mixture of Gaussians, which we take to be 
of the form 

p(x|@) = (1 — w)N(x|6, I) + wN (x|0, al) (10.209) 


where w is the proportion of background clutter and is assumed to be known. The 
prior over @ is taken to be Gaussian 


p(0) = N (0]0, bI) (10.210) 
and Minka (2001a) chooses the parameter values a = 10, b = 100 and w = 0.5. 
The joint distribution of N observations D = {x1,...,x~} and @ is given by 
N 
p(D, 8) = p(0) | | pnl6) (10.211) 
n=1 


and so the posterior distribution comprises a mixture of 2% Gaussians. Thus the 
computational cost of solving this problem exactly would grow exponentially with 
the size of the data set, and so an exact solution is intractable for moderately large 
N. 

To apply EP to the clutter problem, we first identify the factors fo(@) = p(@) 
and fn(0) = p(xn|0). Next we select an approximating distribution from the expo- 
nential family, and for this example it is convenient to choose a spherical Gaussian 


(0) = N(0|m, v1). (10.212) 
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The factor approximations will therefore take the form of exponential-quadratic 
functions of the form a 
fr(O) = Ssn N (O|mn, vnI) (10.213) 


where n = 1,..., N, and we set fo(0) equal to the prior p(@). Note that the use of 
N (0|, -) does not imply that the right-hand side is a well-defined Gaussian density 
(in fact, as we shall see, the variance parameter v,, can be negative) but is simply a 
convenient shorthand notation. The approximations f,,(@), for n = 1,..., N, can 
be initialized to unity, corresponding to Sn = (27Un)? /2 un — œ andm, = 0, 
where D is the dimensionality of x and hence of 0. The initial g(@), defined by 
(10.191), is therefore equal to the prior. 

We then iteratively refine the factors by taking one factor f,(@) at a time and 
applying (10.205), (10.206), and (10.207). Note that we do not need to revise the 
term fo(@) because an EP update will leave this term unchanged. Here we state the 
results and leave the reader to fill in the details. 

First we remove the current estimate f„(0) from q(@) by division using (10.205) 
to give q\" (0), which has mean and inverse variance given by 


m” = m+v\"vZ'(m—m,) (10.214) 
(aT = pty, (10.215) 

Next we evaluate the normalization constant Z,, using (10.206) to give 
Zn = (1 — w)N(xp|m\”, (v\” + 1)1) + wNV(x,|0, al). (10.216) 


Similarly, we compute the mean and variance of q”°™ (0) by finding the mean and 
variance of q\"(0) fn (0) to give 


\n 
m = m+ E n m`”) (10.217) 
\n)2 \ny2 — m\n|}2 
Vv Vv Pn pr +d j Pnl Pn) D(v\n 7 1? (10. ) 
where the quantity 
iat ZN %nl0, al) (10.219) 


has a simple interpretation as the probability of the point x,, not being clutter. Then 
we use (10.207) to compute the refined factor f,,(@) whose parameters are given by 


vat = (wY) — (o)! (10.220) 

m, = mV + (vn +u) (mY — m\*) (10.221) 
Zn 

Sn (10.222) 


(27Un)?/2N (m,y|m\”, (vn + v\")T) 


This refinement process is repeated until a suitable termination criterion is satisfied, 
for instance that the maximum change in parameter values resulting from a complete 
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Figure 10.16 Examples of the approximation of specific factors for a one-dimensional version of the clutter 
problem, showing fn (0) in blue, fn (0) in red, and q`" (0) in green. Notice that the current form for q`” (0) controls 
the range of @ over which fn(0) will be a good approximation to f» (0). 


pass through all factors is less than some threshold. Finally, we use (10.208) to 
evaluate the approximation to the model evidence, given by 


N 
p(D) ~ (2rv”™)P/? exp(B/2) | [ {sn(2rvn)~?/?} (10.223) 


n=1 
where 


B= 


new \T,,,new N 
(cae pees (10.224) 


v 


Examples factor approximations for the clutter problem with a one-dimensional pa- 
rameter space 0 are shown in Figure 10.16. Note that the factor approximations can 
have infinite or even negative values for the ‘variance’ parameter v,,. This simply 
corresponds to approximations that curve upwards instead of downwards and are not 
necessarily problematic provided the overall approximate posterior q(0) has posi- 
tive variance. Figure 10.17 compares the performance of EP with variational Bayes 
(mean field theory) and the Laplace approximation on the clutter problem. 


10.7.2 Expectation propagation on graphs 


So far in our general discussion of EP, we have allowed the factors f;(@) in the 
distribution p(@) to be functions of all of the components of 0, and similarly for the 
approximating factors f(@) in the approximating distribution q(0). We now consider 
situations in which the factors depend only on subsets of the variables. Such restric- 
tions can be conveniently expressed using the framework of probabilistic graphical 
models, as discussed in Chapter 8. Here we use a factor graph representation because 
this encompasses both directed and undirected graphs. 
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Figure 10.17 Comparison of expectation propagation, variational inference, and the Laplace approximation on 
the clutter problem. The left-hand plot shows the error in the predicted posterior mean versus the number of 
floating point operations, and the right-hand plot shows the corresponding results for the model evidence. 
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We shall focus on the case in which the approximating distribution is fully fac- 
torized, and we shall show that in this case expectation propagation reduces to loopy 
belief propagation (Minka, 2001a). To start with, we show this in the context of a 
simple example, and then we shall explore the general case. 

First of all, recall from (10.17) that if we minimize the Kullback-Leibler diver- 
gence KL(p||q) with respect to a factorized distribution q, then the optimal solution 
for each factor is simply the corresponding marginal of p. 

Now consider the factor graph shown on the left in Figure 10.18, which was 
introduced earlier in the context of the sum-product algorithm. The joint distribution 
is given by 

p(x) = Fol Big £2) fo(£2, £3) fel£2, £4). (10.225) 


We seek an approximation q(x) that has the same factorization, so that 


q(x) x Jalt1, £2) fol£2, £3) fel£2, £4). (10.226) 


Note that normalization constants have been omitted, and these can be re-instated at 
the end by local normalization, as is generally done in belief propagation. Now sup- 
pose we restrict attention to approximations in which the factors themselves factorize 
with respect to the individual variables so that 


q(x) x Falei) faz (x2) foo(£2) fos (£3) fe2(£2) fea (T4) (10.227) 


which corresponds to the factor graph shown on the right in Figure 10.18. Because 
the individual factors are factorized, the overall distribution q(x) is itself fully fac- 
torized. 

Now we apply the EP algorithm using the fully factorized approximation. Sup- 
pose that we have initialized all of the factors and that we choose to refine factor 
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Figure 10.18 On the left is a simple factor graph from Figure 8.51 and reproduced here for convenience. On 
the right is the corresponding factorized approximation. 


folx, z3) = fo2(x2) foslx3). We first remove this factor from the approximating 
distribution to give 


q(x) = far (#1) faa (a2) fe2(£2) fea (21) (10.228) 


and we then multiply this by the exact factor f,(x2, x3) to give 


p(x) = q(x) fo(a2, £3) = Fal) fa2l£2)fe2(£2) foa(@a) fol£2, z3). (10.229) 


We now find q™°™ (x) by minimizing the Kullback-Leibler divergence KL(p||q"°™). 
The result, as noted above, is that q™°™ (z) comprises the product of factors, one for 
each variable x;, in which each factor is given by the corresponding marginal of 
p(x). These four marginals are given by 


Blt1) « falz) (10.230) 
Dae) x fa2(a2) fea(ae) > folae, 23) (10.231) 
P(t3) x Y (lez, 8) Faa 2) Feala) } (10.232) 
Dla) x  fea(ara) (10.233) 


and q"°“(x) is obtained by multiplying these marginals together. We see that the 
only factors in q(x) that change when we update folZ2, x3) are those that involve 
the variables in fẹ namely x2 and x3. To obtain the refined factor hlz, z3) = 
fo2(@2) fos (as) we simply divide q™°¥ (x) by q? (x), which gives 


folz) x XO fo(r2, T3) (10.234) 


fos(a3) œ Y (folz, 23) faa(22)fea(22) } (10.235) 


T2 
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These are precisely the messages obtained using belief propagation in which mes- 
sages from variable nodes to factor nodes have been folded into the messages from 
factor nodes to variable nodes. In particular, fp2(x2) corresponds to the message 
Lf, (#2) sent by factor node f, to variable node x2 and is given by (8.81). Simi- 
larly, if we substitute (8.78) into (8.79), we obtain (10.235) in which falza) corre- 
sponds to Hf, —x(£2) and fe2(x2) corresponds to ju f,+2. (#2), giving the message 
fea (x3) which corresponds to Hf, +25 (23). 

This result differs slightly from standard belief propagation in that messages are 
passed in both directions at the same time. We can easily modify the EP procedure 
to give the standard form of the sum-product algorithm by updating just one of the 


factors at a time, for instance if we refine only Foals), then fo2(Z2) is unchanged 


by definition, while the refined version of fp3(a3) is again given by (10.235). If 
we are refining only one term at a time, then we can choose the order in which the 
refinements are done as we wish. In particular, for a tree-structured graph we can 
follow a two-pass update scheme, corresponding to the standard belief propagation 
schedule, which will result in exact inference of the variable and factor marginals. 
The initialization of the approximation factors in this case is unimportant. 

Now let us consider a general factor graph corresponding to the distribution 


=] [40 (10.236) 


where 0; represents the subset of variables associated with factor fi. We approximate 
this using a fully factorized distribution of the form 


0) x [TI] J fix (Oe) (10.237) 
i k 


where @;, corresponds to an individual variable node. Suppose that we wish to refine 
the particular term f;;(6;) keeping all other terms fixed. We first remove the term 


F;(9;) from q(@) to give 
5(0) x [ [TJ fix (x) (10.238) 
ižj k 
and then multiply by the exact factor f;(0@;). To determine the refined term Falh), 


we need only consider the functional ce pendence on 6), and so we simply find the 
corresponding marginal of 


q“ (8) f;(0;). (10.239) 
Up to a multiplicative constant, this involves taking the marginal of f; (0;) multiplied 
by any terms from q (0) that are functions of any of the variables in 0}. Terms that 


correspond to other factors F (0;) for i A j will cancel between numerator and 
denominator when we subsequently divide by q\’(@). We therefore obtain 


hox X Hen me (10.240) 


Omi E85 k mÆl 
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We recognize this as the sum-product rule in the form in which messages from vari- 
able nodes to factor nodes have been eliminated, as illustrated by the example shown 
in Figure 8.50. The quantity fjm(9m) corresponds to the message Hf —0n (0m), 
which factor node j sends to variable node m, and the product over k in (10.240) 
is over all factors that depend on the variables @,, that have variables (other than 
variable @;) in common with factor f;(0;). In other words, to compute the outgoing 
message from a factor node, we take the product of all the incoming messages from 
other factor nodes, multiply by the local factor, and then marginalize. 

Thus, the sum-product algorithm arises as a special case of expectation propa- 
gation if we use an approximating distribution that is fully factorized. This suggests 
that more flexible approximating distributions, corresponding to partially discon- 
nected graphs, could be used to achieve higher accuracy. Another generalization is 
to group factors f;(0;) together into sets and to refine all the factors in a set together 
at each iteration. Both of these approaches can lead to improvements in accuracy 
(Minka, 2001b). In general, the problem of choosing the best combination of group- 
ing and disconnection is an open research issue. 

We have seen that variational message passing and expectation propagation op- 
timize two different forms of the Kullback-Leibler divergence. Minka (2005) has 
shown that a broad range of message passing algorithms can be derived from a com- 
mon framework involving minimization of members of the alpha family of diver- 
gences, given by (10.19). These include variational message passing, loopy belief 
propagation, and expectation propagation, as well as a range of other algorithms, 
which we do not have space to discuss here, such as tree-reweighted message pass- 
ing (Wainwright et al., 2005), fractional belief propagation (Wiegerinck and Heskes, 
2003), and power EP (Minka, 2004). 


œ FY Verify that the log marginal distribution of the observed data In p(X) 
can be decomposed into two terms in the form (10.2) where £(q) is given by (10.3) 
and KL(q||p) is given by (10.4). 


(x) Use the properties E[z,] = mı and E[z2] = mz to solve the simultaneous equa- 
tions (10.13) and (10.15), and hence show that, provided the original distribution 
p(z) is nonsingular, the unique solution for the means of the factors in the approxi- 
mation distribution is given by E[z,] = pı and E[z2] = u2. 


(xx) www | Consider a factorized variational distribution q(Z) of the form (10.5). 
By using the technique of Lagrange multipliers, verify that minimization of the 
Kullback-Leibler divergence KL(p||q) with respect to one of the factors q;(Z;), 
keeping all other factors fixed, leads to the solution (10.17). 


(xx) Suppose that p(x) is some fixed distribution and that we wish to approximate 
it using a Gaussian distribution g(x) = N (x|u, X). By writing down the form of 
the KL divergence KL(p||q) for a Gaussian q(x) and then differentiating, show that 
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10.5 


10.6 


10.7 


10.8 


10.9 


10.10 


10.11 


10.12 


minimization of KL(p||q) with respect to u and ¥ leads to the result that yz is given 
by the expectation of x under p(x) and that X is given by the covariance. 


(« x) www | Consider a model in which the set of all hidden stochastic variables, de- 
noted collectively by Z, comprises some latent variables z together with some model 
parameters 0. Suppose we use a variational distribution that factorizes between la- 
tent variables and parameters so that q(z, 0) = qz(z)q@(@), in which the distribution 
qo(@) is approximated by a point estimate of the form qg(@) = 5(@ — 8) where 85 
is a vector of free parameters. Show that variational optimization of this factorized 
distribution is equivalent to an EM algorithm, in which the E step optimizes q,(z), 
and the M step maximizes the expected complete-data log posterior distribution of 0 
with respect to 0o. 


(xx) The alpha family of divergences is defined by (10.19). Show that the Kullback- 
Leibler divergence KL(p||q) corresponds to a — 1. This can be done by writing 
p° = exp(elnp) = 1 + cln p + O(e?) and then taking € — 0. Similarly show that 
KL(gq||p) corresponds to a — —1. 


(xx) Consider the problem of inferring the mean and precision of a univariate Gaus- 
sian using a factorized variational approximation, as considered in Section 10.1.3. 
Show that the factor g,,(j) is a Gaussian of the form M (u| un, Ag) with mean and 
precision given by (10.26) and (10.27), respectively. Similarly show that the factor 
q,(T) is a gamma distribution of the form Gam(t|ay, by) with parameters given by 
(10.29) and (10.30). 


(x) Consider the variational posterior distribution for the precision of a univariate 
Gaussian whose parameters are given by (10.29) and (10.30). By using the standard 
results for the mean and variance of the gamma distribution given by (B.27) and 
(B.28), show that if we let N — oc, this variational posterior distribution has a 
mean given by the inverse of the maximum likelihood estimator for the variance of 
the data, and a variance that goes to zero. 


(xx) By making use of the standard result E[r] = ay /by for the mean of a gamma 
distribution, together with (10.26), (10.27), (10.29), and (10.30), derive the result 
(10.33) for the reciprocal of the expected precision in the factorized variational treat- 
ment of a univariate Gaussian. 


(x) EMA Derive the decomposition given by (10.34) that is used to find approxi- 
mate posterior distributions over models using variational inference. 


œ» EY By using a Lagrange multiplier to enforce the normalization constraint 
on the distribution q(m), show that the maximum of the lower bound (10.35) is given 
by (10.36). 


(xx) Starting from the joint distribution (10.41), and applying the general result 
(10.9), show that the optimal variational distribution q*(Z) over the latent variables 
for the Bayesian mixture of Gaussians is given by (10.48) by verifying the steps 
given in the text. 
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10.14 
10.15 


10.16 


10.17 


10.18 


10.19 


10.20 
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(xx) www | Starting from (10.54), derive the result (10.59) for the optimum vari- 
ational posterior distribution over u, and Ax in the Bayesian mixture of Gaussians, 
and hence verify the expressions for the parameters of this distribution given by 
(10.60)-(10.63). 


(xx) Using the distribution (10.59), verify the result (10.64). 


(x) Using the result (B.17), show that the expected value of the mixing coefficients 
in the variational mixture of Gaussians is given by (10.69). 


œx» PQQ Verify the results (10.71) and (10.72) for the first two terms in the 
lower bound for the variational Gaussian mixture model given by (10.70). 


(xxx) Verify the results (10.73)-(10.77) for the remaining terms in the lower bound 
for the variational Gaussian mixture model given by (10.70). 


(xxx) In this exercise, we shall derive the variational re-estimation equations for 
the Gaussian mixture model by direct differentiation of the lower bound. To do this 
we assume that the variational distribution has the factorization defined by (10.42) 
and (10.55) with factors given by (10.48), (10.57), and (10.59). Substitute these into 
(10.70) and hence obtain the lower bound as a function of the parameters of the varia- 
tional distribution. Then, by maximizing the bound with respect to these parameters, 
derive the re-estimation equations for the factors in the variational distribution, and 
show that these are the same as those obtained in Section 10.2.1. 


(xx) Derive the result (10.81) for the predictive distribution in the variational treat- 
ment of the Bayesian mixture of Gaussians model. 


(« x) EANA This exercise explores the variational Bayes solution for the mixture of 
Gaussians model when the size N of the data set is large and shows that it reduces (as 
we would expect) to the maximum likelihood solution based on EM derived in Chap- 
ter 9. Note that results from Appendix B may be used to help answer this exercise. 
First show that the posterior distribution q*(A;) of the precisions becomes sharply 
peaked around the maximum likelihood solution. Do the same for the posterior dis- 
tribution of the means q*(f1;,|A,). Next consider the posterior distribution q*(7) 
for the mixing coefficients and show that this too becomes sharply peaked around 
the maximum likelihood solution. Similarly, show that the responsibilities become 
equal to the corresponding maximum likelihood values for large N, by making use 
of the following asymptotic result for the digamma function for large x 


W(x) = mg +O (1/72). (10.241) 


Finally, by making use of (10.80), show that for large N, the predictive distribution 
becomes a mixture of Gaussians. 


(x) Show that the number of equivalent parameter settings due to interchange sym- 
metries in a mixture model with K components is K!. 
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10.22 


10.23 


10.24 


10.25 


10.26 


10.27 


10.28 


(xx) We have seen that each mode of the posterior distribution in a Gaussian mix- 
ture model is a member of a family of K! equivalent modes. Suppose that the result 
of running the variational inference algorithm is an approximate posterior distribu- 
tion q that is localized in the neighbourhood of one of the modes. We can then 
approximate the full posterior distribution as a mixture of K! such q distributions, 
once centred on each mode and having equal mixing coefficients. Show that if we 
assume negligible overlap between the components of the q mixture, the resulting 
lower bound differs from that for a single component q distribution through the ad- 
dition of an extra term In K!. 


(« x) www | Consider a variational Gaussian mixture model in which there is no 
prior distribution over mixing coefficients {7x }. Instead, the mixing coefficients are 
treated as parameters, whose values are to be found by maximizing the variational 
lower bound on the log marginal likelihood. Show that maximizing this lower bound 
with respect to the mixing coefficients, using a Lagrange multiplier to enforce the 
constraint that the mixing coefficients sum to one, leads to the re-estimation result 
(10.83). Note that there is no need to consider all of the terms in the lower bound but 
only the dependence of the bound on the {7 }. 


œ» FY We have seen in Section 10.2 that the singularities arising in the max- 
imum likelihood treatment of Gaussian mixture models do not arise in a Bayesian 
treatment. Discuss whether such singularities would arise if the Bayesian model 
were solved using maximum posterior (MAP) estimation. 


(xx) The variational treatment of the Bayesian mixture of Gaussians, discussed in 
Section 10.2, made use of a factorized approximation (10.5) to the posterior distribu- 
tion. As we saw in Figure 10.2, the factorized assumption causes the variance of the 
posterior distribution to be under-estimated for certain directions in parameter space. 
Discuss qualitatively the effect this will have on the variational approximation to the 
model evidence, and how this effect will vary with the number of components in 
the mixture. Hence explain whether the variational Gaussian mixture will tend to 
under-estimate or over-estimate the optimal number of components. 


(xxx) Extend the variational treatment of Bayesian linear regression to include 
a gamma hyperprior Gam((3|co, do) over 8 and solve variationally, by assuming a 
factorized variational distribution of the form g(w)q(a)q(). Derive the variational 
update equations for the three factors in the variational distribution and also obtain 
an expression for the lower bound and for the predictive distribution. 


(xx) By making use of the formulae given in Appendix B show that the variational 
lower bound for the linear basis function regression model, defined by (10.107), can 
be written in the form (10.107) with the various terms defined by (10.108)—(10.112). 


(xxx) Rewrite the model for the Bayesian mixture of Gaussians, introduced in 
Section 10.2, as a conjugate model from the exponential family, as discussed in 
Section 10.4. Hence use the general results (10.115) and (10.119) to derive the 
specific results (10.48), (10.57), and (10.59). 
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(x) FQ Show that the function f(x) = ln(x) is concave for 0 < x < co 
by computing its second derivative. Determine the form of the dual function g(A) 
defined by (10.133), and verify that minimization of Ax — g(A) with respect to A 
according to (10.132) indeed recovers the function ln(x). 


(x) By evaluating the second derivative, show that the log logistic function f(x) = 
—In(1 + e~*) is concave. Derive the variational upper bound (10.137) directly by 
making a second order Taylor expansion of the log logistic function around a point 


C=. 


(xx) By finding the second derivative with respect to x, show that the function 
f(x) = —In(e*/? + e~*/?) is a concave function of x. Now consider the second 
derivatives with respect to the variable x? and hence show that it is a convex function 
of x”. Plot graphs of f(x) against x and against x”. Derive the lower bound (10.144) 
on the logistic sigmoid function directly by making a first order Taylor series expan- 
sion of the function f(x) in the variable x? centred on the value €?. 


œ» [QTY Consider the variational treatment of logistic regression with sequen- 
tial learning in which data points are arriving one at a time and each must be pro- 
cessed and discarded before the next data point arrives. Show that a Gaussian ap- 
proximation to the posterior distribution can be maintained through the use of the 
lower bound (10.151), in which the distribution is initialized using the prior, and as 
each data point is absorbed its corresponding variational parameter €,, is optimized. 


(x) By differentiating the quantity Q(€, €°') defined by (10.161) with respect to 
the variational parameter €,, show that the update equation for €,, for the Bayesian 
logistic regression model is given by (10.163). 


(xx) In this exercise we derive re-estimation equations for the variational parame- 
ters € in the Bayesian logistic regression model of Section 4.5 by direct maximization 
of the lower bound given by (10.164). To do this set the derivative of £(€) with re- 
spect to €,, equal to zero, making use of the result (3.117) for the derivative of the log 
of a determinant, together with the expressions (10.157) and (10.158) which define 
the mean and covariance of the variational posterior distribution q(w). 


(xx) Derive the result (10.164) for the lower bound £(&) in the variational logistic 
regression model. This is most easily done by substituting the expressions for the 
Gaussian prior g(w) = M (w|mo, So), together with the lower bound h(w, €) on 
the likelihood function, into the integral (10.159) which defines £(€). Next gather 
together the terms which depend on w in the exponential and complete the square 
to give a Gaussian integral, which can then be evaluated by invoking the standard 
result for the normalization coefficient of a multivariate Gaussian. Finally take the 
logarithm to obtain (10.164). 


(xx) Consider the ADF approximation scheme discussed in Section 10.7, and show 
that inclusion of the factor f;(@) leads to an update of the model evidence of the 
form 
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10.37 


10.38 


10.39 


where Z; is the normalization constant defined by (10.197). By applying this result 
recursively, and initializing with po(D) = 1, derive the result 


p(D) = | [Z (10.243) 
J 


(x) EQ Consider the expectation propagation algorithm from Section 10.7, and 
suppose that one of the factors fọ(0) in the definition (10.188) has the same expo- 
nential family functional form as the approximating distribution g(@). Show that if 
the factor fo(@) is initialized to be fo(@), then an EP update to refine fo(@) leaves 
fo(@) unchanged. This situation typically arises when one of the factors is the prior 
p(@), and so we see that the prior factor can be incorporated once exactly and does 
not need to be refined. 


(xxx) In this exercise and the next, we shall verify the results (10.214)-(10.224) 
for the expectation propagation algorithm applied to the clutter problem. Begin by 
using the division formula (10.205) to derive the expressions (10.214) and (10.215) 
by completing the square inside the exponential to identify the mean and variance. 
Also, show that the normalization constant Z,,, defined by (10.206), is given for the 
clutter problem by (10.216). This can be done by making use of the general result 
(2.115). 


(xxx) Show that the mean and variance of q”°Y (0) for EP applied to the clutter 
problem are given by (10.217) and (10.218). To do this, first prove the following 
results for the expectations of @ and 007 under q”°™ (0) 


EJO] = m`” +v\"V min ln Zn (10.244) 
EAT] = 2(v\")?V,\n n Zn + 2E[6]™m\” — |m”? (10.245) 


and then make use of the result (10.216) for Zn. Next, prove the results (10.220)- 
(10.222) by using (10.207) and completing the square in the exponential. Finally, 
use (10.208) to derive the result (10.223). 


For most probabilistic models of practical interest, exact inference is intractable, and 
so we have to resort to some form of approximation. In Chapter 10, we discussed 
inference algorithms based on deterministic approximations, which include methods 
such as variational Bayes and expectation propagation. Here we consider approxi- 
mate inference methods based on numerical sampling, also known as Monte Carlo 
techniques. 

Although for some applications the posterior distribution over unobserved vari- 
ables will be of direct interest in itself, for most situations the posterior distribution 
is required primarily for the purpose of evaluating expectations, for example in order 
to make predictions. The fundamental problem that we therefore wish to address in 
this chapter involves finding the expectation of some function f(z) with respect to a 
probability distribution p(z). Here, the components of z might comprise discrete or 
continuous variables or some combination of the two. Thus in the case of continuous 


523 


524 11. SAMPLING METHODS 


Figure 11.1 


Exercise 11.1 


Schematic illustration of a function f(z) 
whose expectation is to be evaluated with p(z) f(z) 
respect to a distribution p(z). 


XR 


variables, we wish to evaluate the expectation 


ELF] = f f(2)p(2) dz (LA) 


where the integral is replaced by summation in the case of discrete variables. This 
is illustrated schematically for a single continuous variable in Figure 11.1. We shall 
suppose that such expectations are too complex to be evaluated exactly using analyt- 
ical techniques. 

The general idea behind sampling methods is to obtain a set of samples z 
(where l = 1,..., L) drawn independently from the distribution p(z). This allows 
the expectation (11.1) to be approximated by a finite sum 


L 
> 1 
= (1) 
f=7> fe). (11.2) 
I=1 
As long as the samples z are drawn from the distribution p(z), then E[f] = E[S] 


and so the estimator f has the correct mean. The variance of the estimator is given 
by 


A 


var[f] = E [(F- ELA)" (11.3) 


is the variance of the function f(z) under the distribution p(z). It is worth emphasiz- 
ing that the accuracy of the estimator therefore does not depend on the dimension- 
ality of z, and that, in principle, high accuracy may be achievable with a relatively 
small number of samples z. In practice, ten or twenty independent samples may 
suffice to estimate an expectation to sufficient accuracy. 

The problem, however, is that the samples {z } might not be independent, and 
so the effective sample size might be much smaller than the apparent sample size. 
Also, referring back to Figure 11.1, we note that if f(z) is small in regions where 
p(z) is large, and vice versa, then the expectation may be dominated by regions 
of small probability, implying that relatively large sample sizes will be required to 
achieve sufficient accuracy. 

For many models, the joint distribution p(z) is conveniently specified in terms 
of a graphical model. In the case of a directed graph with no observed variables, it is 
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straightforward to sample from the joint distribution (assuming that it is possible to 
sample from the conditional distributions at each node) using the following ances- 
tral sampling approach, discussed briefly in Section 8.1.2. The joint distribution is 
specified by 


M 
p(z) = ] | peilpa,) (11.4) 
tel 


where z; are the set of variables associated with node i, and pa; denotes the set of 
variables associated with the parents of node 7. To obtain a sample from the joint 
distribution, we make one pass through the set of variables in the order z,,..., Zag 
sampling from the conditional distributions p(z;|pa;). This is always possible be- 
cause at each step all of the parent values will have been instantiated. After one pass 
through the graph, we will have obtained a sample from the joint distribution. 

Now consider the case of a directed graph in which some of the nodes are in- 
stantiated with observed values. We can in principle extend the above procedure, at 
least in the case of nodes representing discrete variables, to give the following logic 
sampling approach (Henrion, 1988), which can be seen as a special case of impor- 
tance sampling discussed in Section 11.1.4. At each step, when a sampled value is 
obtained for a variable z; whose value is observed, the sampled value is compared 
to the observed value, and if they agree then the sample value is retained and the al- 
gorithm proceeds to the next variable in turn. However, if the sampled value and the 
observed value disagree, then the whole sample so far is discarded and the algorithm 
starts again with the first node in the graph. This algorithm samples correctly from 
the posterior distribution because it corresponds simply to drawing samples from the 
joint distribution of hidden variables and data variables and then discarding those 
samples that disagree with the observed data (with the slight saving of not continu- 
ing with the sampling from the joint distribution as soon as one contradictory value is 
observed). However, the overall probability of accepting a sample from the posterior 
decreases rapidly as the number of observed variables increases and as the number 
of states that those variables can take increases, and so this approach is rarely used 
in practice. 

In the case of probability distributions defined by an undirected graph, there is 
no one-pass sampling strategy that will sample even from the prior distribution with 
no observed variables. Instead, computationally more expensive techniques must be 
employed, such as Gibbs sampling, which is discussed in Section 11.3. 

As well as sampling from conditional distributions, we may also require samples 
from a marginal distribution. If we already have a strategy for sampling from a joint 
distribution p(u, v), then it is straightforward to obtain samples from the marginal 
distribution p(u) simply by ignoring the values for v in each sample. 

There are numerous texts dealing with Monte Carlo methods. Those of partic- 
ular interest from the statistical inference perspective include Chen et al. (2001), 
Gamerman (1997), Gilks et al. (1996), Liu (2001), Neal (1996), and Robert and 
Casella (1999). Also there are review articles by Besag et al. (1995), Brooks (1998), 
Diaconis and Saloff-Coste (1998), Jerrum and Sinclair (1996), Neal (1993), Tierney 
(1994), and Andrieu ef al. (2003) that provide additional information on sampling 
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11.1. 


Exercise 11.2 


methods for statistical inference. 

Diagnostic tests for convergence of Markov chain Monte Carlo algorithms are 
summarized in Robert and Casella (1999), and some practical guidance on the use of 
sampling methods in the context of machine learning is given in Bishop and Nabney 
(2008). 


Basic Sampling Algorithms 


In this section, we consider some simple strategies for generating random samples 
from a given distribution. Because the samples will be generated by a computer 
algorithm they will in fact be pseudo-random numbers, that is, they will be deter- 
ministically calculated, but must nevertheless pass appropriate tests for randomness. 
Generating such numbers raises several subtleties (Press et al., 1992) that lie outside 
the scope of this book. Here we shall assume that an algorithm has been provided 
that generates pseudo-random numbers distributed uniformly over (0, 1), and indeed 
most software environments have such a facility built in. 


11.1.1 Standard distributions 


We first consider how to generate random numbers from simple nonuniform dis- 
tributions, assuming that we already have available a source of uniformly distributed 
random numbers. Suppose that z is uniformly distributed over the interval (0,1), 
and that we transform the values of z using some function f(-) so that y = f(z). 
The distribution of y will be governed by 


dz 


a (11.5) 


p(y) = p(z) 


where, in this case, p(z) = 1. Our goal is to choose the function f(z) such that the 
resulting values of y have some specific desired distribution p(y). Integrating (11.5) 
we obtain 


y 
i= f PoE (11.6) 


which is the indefinite integral of p(y). Thus, y = h~'(z), and so we have to 
transform the uniformly distributed random numbers using a function which is the 
inverse of the indefinite integral of the desired distribution. This is illustrated in 
Figure 11.2. 

Consider for example the exponential distribution 


ply) = Aexp(—Ay) (11.7) 


where 0 < y < œœ. In this case the lower limit of the integral in (11.6) is 0, and so 
h(y) = 1 — exp(—Ay). Thus, if we transform our uniformly distributed variable z 
using y = —A~' In(1 — z), then y will have an exponential distribution. 


Figure 11.2 
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Figure 11.3 
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Geometrical interpretation of the trans- 
formation method for generating nonuni- 
formly distributed random numbers. h(y) 
is the indefinite integral of the desired dis- 
tribution p(y). If a uniformly distributed 
random variable z is transformed using 
y = h™} (z), then y will be distributed ac- 
cording to p(y). 


Another example of a distribution to which the transformation method can be 
applied is given by the Cauchy distribution 


1 1 
m1it+y? 


ply) = (11.8) 


In this case, the inverse of the indefinite integral can be expressed in terms of the 
‘tan’ function. 

The generalization to multiple variables is straightforward and involves the Ja- 
cobian of the change of variables, so that 


alz, pouz zm) 


11.9 
O(y1,---, ym) ( ) 


P(y1,---,YmM) = P(Z1,---, 2m) 


As a final example of the transformation method we consider the Box-Muller 
method for generating samples from a Gaussian distribution. First, suppose we gen- 
erate pairs of uniformly distributed random numbers 21, z2 € (—1, 1), which we can 
do by transforming a variable distributed uniformly over (0,1) using z — 2z — 1. 
Next we discard each pair unless it satisfies z? + z5 < 1. This leads to a uniform 
distribution of points inside the unit circle with p(z1, z2) = 1/7, as illustrated in 
Figure 11.3. Then, for each pair z1, z2 we evaluate the quantities 


The Box-Muller method for generating Gaussian dis- 1 
tributed random numbers starts by generating samples 
from a uniform distribution inside the unit circle. 
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1/2 
~21 
y = a( za) (11.10) 
rT 
—2 ln z2 ys 
n = a| (11.11) 


where r? = z? + 23. Then the joint distribution of yı and yz is given by 


alz, z2) 
lyi, y2) 


= ER #12) | eo/a] (11.12) 


and so yı and yz are independent and each has a Gaussian distribution with zero 
mean and unit variance. 

If y has a Gaussian distribution with zero mean and unit variance, then oy + u 
will have a Gaussian distribution with mean jz and variance a”. To generate vector- 
valued variables having a multivariate Gaussian distribution with mean u and co- 
variance X, we can make use of the Cholesky decomposition, which takes the form 
£ = LL? (Press et al., 1992). Then, if z is a vector valued random variable whose 
components are independent and Gaussian distributed with zero mean and unit vari- 
ance, then y = u + Lz will have mean p and covariance X. 

Obviously, the transformation technique depends for its success on the ability 
to calculate and then invert the indefinite integral of the required distribution. Such 
operations will only be feasible for a limited number of simple distributions, and so 
we must turn to alternative approaches in search of a more general strategy. Here 
we consider two techniques called rejection sampling and importance sampling. Al- 
though mainly limited to univariate distributions and thus not directly applicable to 
complex problems in many dimensions, they do form important components in more 
general strategies. 


ply, y2) = P(A, 22) 


11.1.2 Rejection sampling 


The rejection sampling framework allows us to sample from relatively complex 
distributions, subject to certain constraints. We begin by considering univariate dis- 
tributions and discuss the extension to multiple dimensions subsequently. 

Suppose we wish to sample from a distribution p(z) that is not one of the simple, 
standard distributions considered so far, and that sampling directly from p(z) is dif- 
ficult. Furthermore suppose, as is often the case, that we are easily able to evaluate 
p(z) for any given value of z, up to some normalizing constant Z, so that 


p(z) = = Pl) (11.13) 
where p(z) can readily be evaluated, but Z, is unknown. 


In order to apply rejection sampling, we need some simpler distribution q(z), 
sometimes called a proposal distribution, from which we can readily draw samples. 


Figure 11.4 
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In the rejection sampling method, kq(z 
samples are drawn from a sim- kq(z0) alz) 
ple distribution q(z) and rejected 
if they fall in the grey area be- 
tween the unnormalized distribu- 
tion p(z) and the scaled distribu- 
tion kq(z). The resulting samples 
are distributed according to p(z), 
which is the normalized version of 


p(z). Zo zZ 


We next introduce a constant k whose value is chosen such that kq(z) > p(z) for 
all values of z. The function kq(z) is called the comparison function and is illus- 
trated for a univariate distribution in Figure 11.4. Each step of the rejection sampler 
involves generating two random numbers. First, we generate a number zo from the 
distribution q(z). Next, we generate a number wo from the uniform distribution over 
[0, kq(zo)]. This pair of random numbers has uniform distribution under the curve 
of the function kq(z). Finally, if uo > p(zo) then the sample is rejected, otherwise 
uo is retained. Thus the pair is rejected if it lies in the grey shaded region in Fig- 
ure 11.4. The remaining pairs then have uniform distribution under the curve of p(z), 
and hence the corresponding z values are distributed according to p(z), as desired. 

The original values of z are generated from the distribution q(z), and these sam- 
ples are then accepted with probability p(z)/kq(z), and so the probability that a 
sample will be accepted is given by 


pee = | {(B(2)/ka(2)} a(z) dz 


TEG dz. (11.14) 


Thus the fraction of points that are rejected by this method depends on the ratio of 
the area under the unnormalized distribution p(z) to the area under the curve kq(z). 
We therefore see that the constant k should be as small as possible subject to the 
limitation that kq(z) must be nowhere less than p(z). 

As an illustration of the use of rejection sampling, consider the task of sampling 
from the gamma distribution 


b*z*~! exp(—bz) 


Gam(z|a, b) = T(a) 


(11.15) 


which, for a > 1, has a bell-shaped form, as shown in Figure 11.5. A suitable 
proposal distribution is therefore the Cauchy (11.8) because this too is bell-shaped 
and because we can use the transformation method, discussed earlier, to sample from 
it. We need to generalize the Cauchy slightly to ensure that it nowhere has a smaller 
value than the gamma distribution. This can be achieved by transforming a uniform 
random variable y using z = btan y + c, which gives random numbers distributed 
according to. 


530 11. SAMPLING METHODS 


Figure 11.5 Plot showing the gamma distribu- 


Exercise 11.9 


Figure 11.6 


0.15 


tion given by (11.15) as the green 
curve, with a scaled Cauchy pro- 
posal distribution shown by the red 
curve. Samples from the gamma 0.1 
distribution can be obtained by 

sampling from the Cauchy and p(z) 
then applying the rejection sam- 

pling criterion. 0.05 } 


= k 
~ 14+ (z—6)2/b?" 
The minimum reject rate is obtained by setting c = a — 1, b? = 2a — 1 and choos- 


ing the constant k to be as small as possible while still satisfying the requirement 
kq(z) > p(z). The resulting comparison function is also illustrated in Figure 11.5. 


q(z) (11.16) 


11.1.3 Adaptive rejection sampling 


In many instances where we might wish to apply rejection sampling, it proves 
difficult to determine a suitable analytic form for the envelope distribution g(z). An 
alternative approach is to construct the envelope function on the fly based on mea- 
sured values of the distribution p(z) (Gilks and Wild, 1992). Construction of an 
envelope function is particularly straightforward for cases in which p(z) is log con- 
cave, in other words when In p(z) has derivatives that are nonincreasing functions 
of z. The construction of a suitable envelope function is illustrated graphically in 
Figure 11.6. 

The function In p(z) and its gradient are evaluated at some initial set of grid 
points, and the intersections of the resulting tangent lines are used to construct the 
envelope function. Next a sample value is drawn from the envelope distribution. 
This is straightforward because the log of the envelope distribution is a succession 


In the case of distributions that are 

log concave, an envelope function In p(z) 

for use in rejection sampling can be 

constructed using the tangent lines Jy IN 
computed at a set of grid points. If a 

sample point is rejected, it is added 

to the set of grid points and used to 

refine the envelope distribution. 


ZL 22 23 z 


Figure 11.7 
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Illustrative example of rejection 


sampling involving sampling from a 0:3 
Gaussian distribution p(z) shown by 
the green curve, by using rejection 
' ton A(z) 
sampling from a proposal distri- 
bution q(z) that is also Gaussian 
and whose scaled version kq(z) is 0.25 
shown by the red curve. a 
0 1 
-5 0 Zz 5 


of linear functions, and hence the envelope distribution itself comprises a piecewise 
exponential distribution of the form 


q(z) = kiàiexp {—A;(z — zi-1)} Ži-1 ZS ži- (11.17) 


Once a sample has been drawn, the usual rejection criterion can be applied. If the 
sample is accepted, then it will be a draw from the desired distribution. If, however, 
the sample is rejected, then it is incorporated into the set of grid points, a new tangent 
line is computed, and the envelope function is thereby refined. As the number of 
grid points increases, so the envelope function becomes a better approximation of 
the desired distribution p(z) and the probability of rejection decreases. 

A variant of the algorithm exists that avoids the evaluation of derivatives (Gilks, 
1992). The adaptive rejection sampling framework can also be extended to distri- 
butions that are not log concave, simply by following each rejection sampling step 
with a Metropolis-Hastings step (to be discussed in Section 11.2.2), giving rise to 
adaptive rejection Metropolis sampling (Gilks et al., 1995). 

Clearly for rejection sampling to be of practical value, we require that the com- 
parison function be close to the required distribution so that the rate of rejection is 
kept to a minimum. Now let us examine what happens when we try to use rejection 
sampling in spaces of high dimensionality. Consider, for the sake of illustration, 
a somewhat artificial problem in which we wish to sample from a zero-mean mul- 
tivariate Gaussian distribution with covariance asl where I is the unit matrix, by 
rejection sampling from a proposal distribution that is itself a zero-mean Gaussian 
distribution having covariance oI. Obviously, we must have o > a in order that 
there exists a k such that kq(z) > p(z). In D-dimensions the optimum value of k 
is given by k = (oq/cp)”, as illustrated for D = 1 in Figure 11.7. The acceptance 
rate will be the ratio of volumes under p(z) and kq(z), which, because both distribu- 
tions are normalized, is just 1/k. Thus the acceptance rate diminishes exponentially 
with dimensionality. Even if o, exceeds o,, by just one percent, for D = 1, 000 the 
acceptance ratio will be approximately 1/20,000. In this illustrative example the 
comparison function is close to the required distribution. For more practical exam- 
ples, where the desired distribution may be multimodal and sharply peaked, it will 
be extremely difficult to find a good proposal distribution and comparison function. 
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Figure 11.8 


Importance sampling addresses the prob- 
lem of evaluating the expectation of a func- 
tion f(z) with respect to a distribution p(z) 
from which it is difficult to draw samples di- 
rectly. Instead, samples {2} are drawn 
from a simpler distribution g(z), and the 
corresponding terms in the summation are 
weighted by the ratios p(z\) /q(z\). 


Furthermore, the exponential decrease of acceptance rate with dimensionality is a 
generic feature of rejection sampling. Although rejection can be a useful technique 
in one or two dimensions it is unsuited to problems of high dimensionality. It can, 
however, play a role as a subroutine in more sophisticated algorithms for sampling 
in high dimensional spaces. 


11.1.4 Importance sampling 


One of the principal reasons for wishing to sample from complicated probability 
distributions is to be able to evaluate expectations of the form (11.1). The technique 
of importance sampling provides a framework for approximating expectations di- 
rectly but does not itself provide a mechanism for drawing samples from distribution 
p(z). 

The finite sum approximation to the expectation, given by (11.2), depends on 
being able to draw samples from the distribution p(z). Suppose, however, that it is 
impractical to sample directly from p(z) but that we can evaluate p(z) easily for any 
given value of z. One simplistic strategy for evaluating expectations would be to 
discretize z-space into a uniform grid and to evaluate the integrand as a sum of the 


form 
L 


EL ~ Dj p2) £2). qais 


l=1 


An obvious problem with this approach is that the number of terms in the summation 
grows exponentially with the dimensionality of z. Furthermore, as we have already 
noted, the kinds of probability distributions of interest will often have much of their 
mass confined to relatively small regions of z space and so uniform sampling will be 
very inefficient because in high-dimensional problems, only a very small proportion 
of the samples will make a significant contribution to the sum. We would really like 
to choose the sample points to fall in regions where p(z) is large, or ideally where 
the product p(z) f(z) is large. 

As in the case of rejection sampling, importance sampling is based on the use 
of a proposal distribution q(z) from which it is easy to draw samples, as illustrated 
in Figure 11.8. We can then express the expectation in the form of a finite sum over 
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samples {z")} drawn from q(z) 


BL = 


K 


L 
1 (1) 
F S iyere.) (11.19) 


The quantities r; = p(z®)/q(z®) are known as importance weights, and they cor- 
rect the bias introduced by sampling from the wrong distribution. Note that, unlike 
rejection sampling, all of the samples generated are retained. 

It will often be the case that the distribution p(z) can only be evaluated up to a 
normalization constant, so that p(z) = p(z)/Z, where p(z) can be evaluated easily, 
whereas Z, is unknown. Similarly, we may wish to use an importance sampling 
distribution g(z) = q(z)/Z,, which has the same property. We then have 


Elf] 


I 
Ga 
= 
Z 
poa 
D 
or 
N 


Zp q(z) 
Bix 
~ ZT dhe). (11.20) 
l=1 


where 7, = p(z) /q(z™). We can use the same sample set to evaluate the ratio 
Zp/Zq With the result 


Zz = z [Poa | Fata) ae 


1 L 
>n (11.21) 
j=1 
and hence 
L 
ELA] = So wf (2) (11.22) 
l=1 
where we have defined 
Tı plz) /q(2) 


wi (11.23) 


Enim om B/E) 


As with rejection sampling, the success of the importance sampling approach 
depends crucially on how well the sampling distribution q(z) matches the desired 
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distribution p(z). If, as is often the case, p(z) f(z) is strongly varying and has a sig- 
nificant proportion of its mass concentrated over relatively small regions of z space, 
then the set of importance weights {r;} may be dominated by a few weights hav- 
ing large values, with the remaining weights being relatively insignificant. Thus the 
effective sample size can be much smaller than the apparent sample size L. The prob- 
lem is even more severe if none of the samples falls in the regions where p(z) f(z) 
is large. In that case, the apparent variances of r; and r; f(z“) may be small even 
though the estimate of the expectation may be severely wrong. Hence a major draw- 
back of the importance sampling method is the potential to produce results that are 
arbitrarily in error and with no diagnostic indication. This also highlights a key re- 
quirement for the sampling distribution g(z), namely that it should not be small or 
zero in regions where p(z) may be significant. 

For distributions defined in terms of a graphical model, we can apply the impor- 
tance sampling technique in various ways. For discrete variables, a simple approach 
is called uniform sampling. The joint distribution for a directed graph is defined 
by (11.4). Each sample from the joint distribution is obtained by first setting those 
variables z; that are in the evidence set equal to their observed values. Each of the 
remaining variables is then sampled independently from a uniform distribution over 
the space of possible instantiations. To determine the corresponding weight associ- 
ated with a sample z“’), we note that the sampling distribution Q(z) is uniform over 
the possible choices for z, and that p(z|x) = p(z), where x denotes the subset of 
variables that are observed, and the equality follows from the fact that every sample 
z that is generated is necessarily consistent with the evidence. Thus the weights r; 
are simply proportional to p(z). Note that the variables can be sampled in any order. 
This approach can yield poor results if the posterior distribution is far from uniform, 
as is often the case in practice. 

An improvement on this approach is called likelihood weighted sampling (Fung 
and Chang, 1990; Shachter and Peot, 1990) and is based on ancestral sampling of 
the variables. For each variable in turn, if that variable is in the evidence set, then it 
is just set to its instantiated value. If it is not in the evidence set, then it is sampled 
from the conditional distribution p(z;|pa,;) in which the conditioning variables are 
set to their currently sampled values. The weighting associated with the resulting 
sample z is then given by 


rz) = |] eee lI a = |[ p@ilpa,). (11.24) 
zige 


Zi|pa; zice Zie 

This method can be further extended using self-importance sampling (Shachter and 
Peot, 1990) in which the importance sampling distribution is continually updated to 
reflect the current estimated posterior distribution. 


11.1.5 Sampling-importance-resampling 


The rejection sampling method discussed in Section 11.1.2 depends in part for 
its success on the determination of a suitable value for the constant k. For many 
pairs of distributions p(z) and q(z), it will be impractical to determine a suitable 
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value for k in that any value that is sufficiently large to guarantee a bound on the 
desired distribution will lead to impractically small acceptance rates. 

As in the case of rejection sampling, the sampling-importance-resampling (SIR) 
approach also makes use of a sampling distribution q(z) but avoids having to de- 
termine the constant k. There are two stages to the scheme. In the first stage, 


L samples z“),...,2z‘) are drawn from q(z). Then in the second stage, weights 
W1,--.,Wz are constructed using (11.23). Finally, a second set of L samples is 
drawn from the discrete distribution (z')),... ,z‘“)) with probabilities given by the 
weights (w1,..., wz). 


The resulting L samples are only approximately distributed according to p(z), 
but the distribution becomes correct in the limit L — oo. To see this, consider the 
univariate case, and note that the cumulative distribution of the resampled values is 
given by 


plz <a) = 5 w 
iz@O<a 
Le t(2 < a)p(z)/a(z) 
di P(2)/q(z) 


where J(.) is the indicator function (which equals 1 if its argument is true and 0 
otherwise). Taking the limit L — oo, and assuming suitable regularity of the dis- 
tributions, we can replace the sums by integrals weighted according to the original 
sampling distribution q(z) 


(11.25) 


I E O 
peed = 
f (Ble) /al2)} a(2) dz 
fie < a)p(z) dz 


= fie < a)p(z) dz (11.26) 


which is the cumulative distribution function of p(z). Again, we see that the normal- 
ization of p(z) is not required. 

For a finite value of L, and a given initial sample set, the resampled values will 
only approximately be drawn from the desired distribution. As with rejection sam- 
pling, the approximation improves as the sampling distribution q(z) gets closer to 
the desired distribution p(z). When q(z) = p(z), the initial samples (z“,... , z) 
have the desired distribution, and the weights w,, = 1/L so that the resampled values 
also have the desired distribution. 

If moments with respect to the distribution p(z) are required, then they can be 
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evaluated directly using the original samples together with the weights, because 


H[f) = f #z)p(a) de 


(11.27) 


N? 
M= 
= 
a 
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11.1.6 Sampling and the EM algorithm 


In addition to providing a mechanism for direct implementation of the Bayesian 
framework, Monte Carlo methods can also play a role in the frequentist paradigm, 
for example to find maximum likelihood solutions. In particular, sampling methods 
can be used to approximate the E step of the EM algorithm for models in which the 
E step cannot be performed analytically. Consider a model with hidden variables 
Z, visible (observed) variables X, and parameters 0. The function that is optimized 
with respect to 0 in the M step is the expected complete-data log likelihood, given 
by 


Q(6,0°") = J zx. 0") mpz, xo) 2, (11.28) 
We can use sampling methods to approximate this integral by a finite sum over sam- 


ples {Z0}, which are drawn from the current estimate for the posterior distribution 
p(Z|X, 0°"), so that 


In p(Z, X|@). (11.29) 


Sie 
Mer 


Q(0, 0°") ~ 


l=1 


The Q function is then optimized in the usual way in the M step. This procedure is 
called the Monte Carlo EM algorithm. 

It is straightforward to extend this to the problem of finding the mode of the 
posterior distribution over 0 (the MAP estimate) when a prior distribution p(@) has 
been defined, simply by adding In p(@) to the function Q(@, 0°") before performing 
the M step. 

A particular instance of the Monte Carlo EM algorithm, called stochastic EM, 
arises if we consider a finite mixture model, and draw just one sample at each E step. 
Here the latent variable Z characterizes which of the K components of the mixture 
is responsible for generating each data point. In the E step, a sample of Z is taken 
from the posterior distribution p(Z|X, 0°") where X is the data set. This effectively 
makes a hard assignment of each data point to one of the components in the mixture. 
In the M step, this sampled approximation to the posterior distribution is used to 
update the model parameters in the usual way. 
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Now suppose we move from a maximum likelihood approach to a full Bayesian 
treatment in which we wish to sample from the posterior distribution over the param- 
eter vector 0. In principle, we would like to draw samples from the joint posterior 
p(@, Z|X), but we shall suppose that this is computationally difficult. Suppose fur- 
ther that it is relatively straightforward to sample from the complete-data parameter 
posterior p(@|Z, X). This inspires the data augmentation algorithm, which alter- 
nates between two steps known as the I-step (imputation step, analogous to an E 
step) and the P-step (posterior step, analogous to an M step). 


IP Algorithm 


I-step. We wish to sample from p(Z|X) but we cannot do this directly. We 
therefore note the relation 


p(Z|X) = f v(zie, xyo(a1x) do (11.30) 


and hence for! = 1,..., L we first draw a sample 9 from the current esti- 
mate for p(@|X), and then use this to draw a sample Z from p(Z|0® , X). 


P-step. Given the relation 
(1X) = | p(0\2, X)p(ZIX) az (11.31) 


we use the samples {Z0 } obtained from the I-step to compute a revised 
estimate of the posterior distribution over 0 given by 


L 
p(O|X) ~ ZH (@|Z, X). (11.32) 


By assumption, it will be feasible to sample from this approximation in the 
I-step. 


Note that we are making a (somewhat artificial) distinction between parameters 0 
and hidden variables Z. From now on, we blur this distinction and focus simply on 
the problem of drawing samples from a given posterior distribution. 


11.2. Markov Chain Monte Carlo 


In the previous section, we discussed the rejection sampling and importance sam- 
pling strategies for evaluating expectations of functions, and we saw that they suffer 
from severe limitations particularly in spaces of high dimensionality. We therefore 
turn in this section to a very general and powerful framework called Markov chain 
Monte Carlo (MCMC), which allows sampling from a large class of distributions, 
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Section 11.2.1 


and which scales well with the dimensionality of the sample space. Markov chain 
Monte Carlo methods have their origins in physics (Metropolis and Ulam, 1949), 
and it was only towards the end of the 1980s that they started to have a significant 
impact in the field of statistics. 

As with rejection and importance sampling, we again sample from a proposal 
distribution. This time, however, we maintain a record of the current state z, and 
the proposal distribution q(z|z‘~)) depends on this current state, and so the sequence 
of samples 2“) ,z),... forms a Markov chain. Again, if we write p(z) = p(z)/Zp, 
we will assume that p(z) can readily be evaluated for any given value of z, although 
the value of Zp may be unknown. The proposal distribution itself is chosen to be 
sufficiently simple that it is straightforward to draw samples from it directly. At 
each cycle of the algorithm, we generate a candidate sample z* from the proposal 
distribution and then accept the sample according to an appropriate criterion. 

In the basic Metropolis algorithm (Metropolis et al., 1953), we assume that the 
proposal distribution is symmetric, that is g(z4|zeB) = q(zZe|za) for all values of 
za and zp. The candidate sample is then accepted with probability 


Near) Sain (1 an (11.33) 


This can be achieved by choosing a random number u with uniform distribution over 
the unit interval (0,1) and then accepting the sample if A(z*,z'7)) > u. Note that 
if the step from z'7) to z* causes an increase in the value of p(z), then the candidate 
point is certain to be kept. 

If the candidate sample is accepted, then z+!) = z*, otherwise the candidate 
point z* is discarded, z‘7+") is set to z(™ and another candidate sample is drawn 
from the distribution q(z|z‘™+!). This is in contrast to rejection sampling, where re- 
jected samples are simply discarded. In the Metropolis algorithm when a candidate 
point is rejected, the previous sample is included instead in the final list of samples, 
leading to multiple copies of samples. Of course, in a practical implementation, 
only a single copy of each retained sample would be kept, along with an integer 
weighting factor recording how many times that state appears. As we shall see, as 
long as q(z4|Zz) is positive for any values of z4 and zp (this is a sufficient but 
not necessary condition), the distribution of z‘7) tends to p(z) as T — oo. It should 
be emphasized, however, that the sequence z‘!),z?),... is not a set of independent 
samples from p(z) because successive samples are highly correlated. If we wish to 
obtain independent samples, then we can discard most of the sequence and just re- 
tain every M* sample. For M sufficiently large, the retained samples will for all 
practical purposes be independent. Figure 11.9 shows a simple illustrative exam- 
ple of sampling from a two-dimensional Gaussian distribution using the Metropolis 
algorithm in which the proposal distribution is an isotropic Gaussian. 

Further insight into the nature of Markov chain Monte Carlo algorithms can be 
gleaned by looking at the properties of a specific example, namely a simple random 


Figure 11.9 A simple illustration using Metropo- 
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lis algorithm to sample from a 
Gaussian distribution whose one 
standard-deviation contour is shown 5 5 | 
by the ellipse. The proposal distribu- 
tion is an isotropic Gaussian distri- 
bution whose standard deviation is ņ [ 
0.2. Steps that are accepted are 
shown as green lines, and rejected 
steps are shown in red. A total of 1.5} 
150 candidate samples are gener- 
ated, of which 43 are rejected. 


walk. Consider a state space z consisting of the integers, with probabilities 


plz) = z0) = 0.5 (11.34) 
p(z7+9 =z) 41) = 0.25 (11.35) 
p(z7+9 = y(t) _ 1) = 0.25 (11.36) 


where z‘7) denotes the state at step 7. If the initial state is z® = 0, then by sym- 
metry the expected state at time 7 will also be zero E[z(™)] = 0, and similarly it is 
easily seen that E[(z‘"))?] = 7/2. Thus after 7 steps, the random walk has only trav- 
elled a distance that on average is proportional to the square root of 7. This square 
root dependence is typical of random walk behaviour and shows that random walks 
are very inefficient in exploring the state space. As we shall see, a central goal in 
designing Markov chain Monte Carlo methods is to avoid random walk behaviour. 


11.2.1 Markov chains 


Before discussing Markov chain Monte Carlo methods in more detail, it is use- 
ful to study some general properties of Markov chains in more detail. In particular, 
we ask under what circumstances will a Markov chain converge to the desired dis- 
tribution. A first-order Markov chain is defined to be a series of random variables 
2, ...,2 such that the following conditional independence property holds for 
me {1,..., M — 1} 


pi2Dig) 20) = p(z |z), (11.37) 
This of course can be represented as a directed graph in the form of a chain, an ex- 


ample of which is shown in Figure 8.38. We can then specify the Markov chain by 
giving the probability distribution for the initial variable p(z) together with the 
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conditional probabilities for subsequent variables in the form of transition probabil- 
ities Tn (z™ ,20™Y)) = p20) 2), A Markov chain is called homogeneous 
if the transition probabilities are the same for all m. 

The marginal probability for a particular variable can be expressed in terms of 
the marginal probability for the previous variable in the chain in the form 


p(z™+9) = >» plat) |z )p(z™). (11.38) 


zim) 


A distribution is said to be invariant, or stationary, with respect to a Markov chain 
if each step in the chain leaves that distribution invariant. Thus, for a homogeneous 
Markov chain with transition probabilities T(z’, z), the distribution p* (z) is invariant 


if 
(2) = > Te zp’). (11.39) 


Note that a given Markov chain may have more than one invariant distribution. For 
instance, if the transition probabilities are given by the identity transformation, then 
any distribution will be invariant. 

A sufficient (but not necessary) condition for ensuring that the required distribu- 
tion p(z) is invariant is to choose the transition probabilities to satisfy the property 
of detailed balance, defined by 


p*(z)T(z,2') = p*(z')T(z',z) (11.40) 


for the particular distribution p*(z). It is easily seen that a transition probability 
that satisfies detailed balance with respect to a particular distribution will leave that 
distribution invariant, because 


X p*(2')T(2',2) = X (2) T(z, 2’) = p*(z) X p(z'|z) =p*(z). (11.41) 


A Markov chain that respects detailed balance is said to be reversible. 

Our goal is to use Markov chains to sample from a given distribution. We can 
achieve this if we set up a Markov chain such that the desired distribution is invariant. 
However, we must also require that for m — oo, the distribution p(z")) converges 
to the required invariant distribution p*(z), irrespective of the choice of initial dis- 
tribution p(z)). This property is called ergodicity, and the invariant distribution 
is then called the equilibrium distribution. Clearly, an ergodic Markov chain can 
have only one equilibrium distribution. It can be shown that a homogeneous Markov 
chain will be ergodic, subject only to weak restrictions on the invariant distribution 
and the transition probabilities (Neal, 1993). 

In practice we often construct the transition probabilities from a set of ‘base’ 
transitions B,,..., Bx. This can be achieved through a mixture distribution of the 
form 


K 
T(z’, z) = X arBk(z', z) (11.42) 
k=1 
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for some set of mixing coefficients a,,...,ax satisfying a, > 0 and ee ap = 1. 
Alternatively, the base transitions may be combined through successive application, 
so that 


=e >) Bila’, 21)... Be-1(@K-2,2K-1)Bx(&x—1,2). (11.43) 


Zn-1 


If a distribution is invariant with respect to each of the base transitions, then obvi- 
ously it will also be invariant with respect to either of the T(z’, z) given by (11.42) 
or (11.43). For the case of the mixture (11.42), if each of the base transitions sat- 
isfies detailed balance, then the mixture transition T will also satisfy detailed bal- 
ance. This does not hold for the transition probability constructed using (11.43), al- 
though by symmetrizing the order of application of the base transitions, in the form 
Bı, B2,..., Bg, Bx,..., B2, By, detailed balance can be restored. A common ex- 
ample of the use of composite transition probabilities is where each base transition 
changes only a subset of the variables. 


11.2.2 The Metropolis-Hastings algorithm 


Earlier we introduced the basic Metropolis algorithm, without actually demon- 
strating that it samples from the required distribution. Before giving a proof, we 
first discuss a generalization, known as the Metropolis-Hastings algorithm (Hast- 
ings, 1970), to the case where the proposal distribution is no longer a symmetric 
function of its arguments. In particular at step 7 of the algorithm, in which the cur- 
rent state is z'"), we draw a sample z* from the distribution q;,(z|z‘™’) and then 
accept it with probability A;(z*,z-) where 


aed ee (T)|y* 
Ag(2*, z0) = min (1,28 hala z) i (11.44) 


P(2Z™ )ag(z*|z0) 


Here k labels the members of the set of possible transitions being considered. Again, 
the evaluation of the acceptance criterion does not require knowledge of the normal- 
izing constant Z, in the probability distribution p(z) = p(z)/Z,. For a symmetric 
proposal distribution the Metropolis-Hastings criterion (11.44) reduces to the stan- 
dard Metropolis criterion given by (11.33). 

We can show that p(z) is an invariant distribution of the Markov chain defined 
by the Metropolis-Hastings algorithm by showing that detailed balance, defined by 
(11.40), is satisfied. Using (11.44) we have 


P(2)qx(zl2')Ag(z',z) = min (p(z)qe(z|2'), p(z")gx(z'|z)) 
min (p(2')qx(2'|2), p(z) qx (z|2")) 
= pl2')qn(2!|2) Ag(z, 2’) (11.45) 


as required. 

The specific choice of proposal distribution can have a marked effect on the 
performance of the algorithm. For continuous state spaces, a common choice is a 
Gaussian centred on the current state, leading to an important trade-off in determin- 
ing the variance parameter of this distribution. If the variance is small, then the 
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Figure 11.10 Schematic illustration of the use of an isotropic 


11.3. 


Gaussian proposal distribution (blue circle) to 
sample from a correlated multivariate Gaussian 
distribution (red ellipse) having very different stan- Tmax 
dard deviations in different directions, using the 
Metropolis-Hastings algorithm. In order to keep 

the rejection rate low, the scale p of the proposal 
distribution should be on the order of the smallest 
standard deviation omin, which leads to random 

walk behaviour in which the number of steps sep- 7 min 
arating states that are approximately independent 

is of order (omax/Omin)” Where omax is the largest 
standard deviation. 


proportion of accepted transitions will be high, but progress through the state space 
takes the form of a slow random walk leading to long correlation times. However, 
if the variance parameter is large, then the rejection rate will be high because, in the 
kind of complex problems we are considering, many of the proposed steps will be 
to states for which the probability p(z) is low. Consider a multivariate distribution 
p(z) having strong correlations between the components of z, as illustrated in Fig- 
ure 11.10. The scale p of the proposal distribution should be as large as possible 
without incurring high rejection rates. This suggests that p should be of the same 
order as the smallest length scale omin. The system then explores the distribution 
along the more extended direction by means of a random walk, and so the number 
of steps to arrive at a state that is more or less independent of the original state is 
of order (Omax/ Omin): In fact in two dimensions, the increase in rejection rate as p 
increases is offset by the larger steps sizes of those transitions that are accepted, and 
more generally for a multivariate Gaussian the number of steps required to obtain 
independent samples scales like (Ox /02)* where o> is the second-smallest stan- 
dard deviation (Neal, 1993). These details aside, it remains the case that if the length 
scales over which the distributions vary are very different in different directions, then 
the Metropolis Hastings algorithm can have very slow convergence. 


Gibbs Sampling 


Gibbs sampling (Geman and Geman, 1984) is a simple and widely applicable Markov 
chain Monte Carlo algorithm and can be seen as a special case of the Metropolis- 
Hastings algorithm. 

Consider the distribution p(z) = p(z1,..., zm ) from which we wish to sample, 
and suppose that we have chosen some initial state for the Markov chain. Each step 
of the Gibbs sampling procedure involves replacing the value of one of the variables 
by a value drawn from the distribution of that variable conditioned on the values of 
the remaining variables. Thus we replace z; by a value drawn from the distribution 
p(z4|Z\;), where z; denotes the it component of z, and Z\; denotes 21,..., Zm but 
with z; omitted. This procedure is repeated either by cycling through the variables 
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in some particular order or by choosing the variable to be updated at each step at 
random from some distribution. 
For example, suppose we have a distribution p(z1, 22, z3) over three variables, 


and at 7 7 of the algorithm we have selected values 2{”, 2” and 2S. We first 


replace Zr ) by anew value ae") obtained by sampling from the conditional distri- 
bution 

piala a h (11.46) 
Next we replace zt ) by a value we ) obtained by sampling from the conditional 
distribution 

p( zat, 2) (11.47) 
so that the new value for z; is used straight away in subsequent sampling steps. Then 
we update z3 with a sample ant) drawn from 

plzgl277, 259) (11.48) 


and so on, cycling through the three variables in turn. 


Gibbs Sampling 


1. Initialize {z; : i = 1,..., M} 
2. For T = l;a; T: 


— Sample fed plalz, 28, ..., 247). 

— Sample 2°") ~ p(zajz tP, 28”, ..., z0). 

- Saale Pa pizt, oe a eee zo). 
— Sample gers) ~ plzm|2*?, 24), z a): 
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To show that this procedure samples from the required distribution, we first of 
all note that the distribution p(z) is an invariant of each of the Gibbs sampling steps 
individually and hence of the whole Markov chain. This follows from the fact that 
when we sample from p(z;|{z\;), the marginal distribution p(z\;) is clearly invariant 
because the value of z\; is unchanged. Also, each step by definition samples from the 
correct conditional distribution p(z;|z\;). Because these conditional and marginal 
distributions together specify the joint distribution, we see that the joint distribution 
is itself invariant. 

The second requirement to be satisfied in order that the Gibbs sampling proce- 
dure samples from the correct distribution is that it be ergodic. A sufficient condition 
for ergodicity is that none of the conditional distributions be anywhere zero. If this 
is the case, then any point in z space can be reached from any other point in a finite 
number of steps involving one update of each of the component variables. If this 
requirement is not satisfied, so that some of the conditional distributions have zeros, 
then ergodicity, if it applies, must be proven explicitly. 

The distribution of initial states must also be specified in order to complete the 
algorithm, although samples drawn after many iterations will effectively become 
independent of this distribution. Of course, successive samples from the Markov 
chain will be highly correlated, and so to obtain samples that are nearly independent 
it will be necessary to subsample the sequence. 

We can obtain the Gibbs sampling procedure as a particular instance of the 
Metropolis-Hastings algorithm as follows. Consider a Metropolis-Hastings sampling 
step involving the variable z4 in which the remaining variables Z\ remain fixed, and 
for which the transition probability from z to z* is given by qx (z*|z) = p(zf|2\x). 
We note that zÝ, = z\; because these components are unchanged by the sampling 
step. Also, p(z) = p(zx|Z\x)p(Z\x). Thus the factor that determines the acceptance 
probability in the Metropolis-Hastings (11.44) is given by 


Meth p(Z*)qr(z|z*) PCZK |Z p) P(A) PZB.) 

i p(z)qg(z*|z) — p(zk|Z\k)p(zk)p(zġ|z\) 

where we have used ZÝ; = Z\,. Thus the Metropolis-Hastings steps are always 
accepted. 

As with the Metropolis algorithm, we can gain some insight into the behaviour of 
Gibbs sampling by investigating its application to a Gaussian distribution. Consider 
a correlated Gaussian in two variables, as illustrated in Figure 11.11, having con- 
ditional distributions of width l and marginal distributions of width L. The typical 
step size is governed by the conditional distributions and will be of order /. Because 
the state evolves according to a random walk, the number of steps needed to obtain 
independent samples from the distribution will be of order (L/1)?. Of course if the 
Gaussian distribution were uncorrelated, then the Gibbs sampling procedure would 
be optimally efficient. For this simple problem, we could rotate the coordinate sys- 
tem in order to decorrelate the variables. However, in practical applications it will 
generally be infeasible to find such transformations. 

One approach to reducing random walk behaviour in Gibbs sampling is called 
over-relaxation (Adler, 1981). In its original form, this applies to problems for which 


=1 (11.49) 


Figure 11.11 
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Illustration of Gibbs sampling by alter- 
nate updates of two variables whose 
distribution is a correlated Gaussian. 
The step size is governed by the stan- 
dard deviation of the conditional distri- 
bution (green curve), and is O(l), lead- 
ing to slow progress in the direction of 
elongation of the joint distribution (red 
ellipse). The number of steps needed 
to obtain an independent sample from 
the distribution is O((L/1)”). 


Zi 


the conditional distributions are Gaussian, which represents a more general class of 
distributions than the multivariate Gaussian because, for example, the non-Gaussian 
distribution p(z, y) x exp(—z*y*) has Gaussian conditional distributions. At each 
step of the Gibbs sampling algorithm, the conditional distribution for a particular 
component z; has some mean p; and some variance o?. In the over-relaxation frame- 
work, the value of z; is replaced with 


z = mi + a(zi — i) + oill a?) v (11.50) 


where v is a Gaussian random variable with zero mean and unit variance, and a 
is a parameter such that —1 < œ < 1. Fora = 0, the method is equivalent to 
standard Gibbs sampling, and for a < 0 the step is biased to the opposite side of the 
mean. This step leaves the desired distribution invariant because if z; has mean p; 
and variance a7, then so too does zj. The effect of over-relaxation is to encourage 
directed motion through state space when the variables are highly correlated. The 
framework of ordered over-relaxation (Neal, 1999) generalizes this approach to non- 
Gaussian distributions. 

The practical applicability of Gibbs sampling depends on the ease with which 
samples can be drawn from the conditional distributions p(z;,|z\,). In the case of 
probability distributions specified using graphical models, the conditional distribu- 
tions for individual nodes depend only on the variables in the corresponding Markov 
blankets, as illustrated in Figure 11.12. For directed graphs, a wide choice of condi- 
tional distributions for the individual nodes conditioned on their parents will lead to 
conditional distributions for Gibbs sampling that are log concave. The adaptive re- 
jection sampling methods discussed in Section 11.1.3 therefore provide a framework 
for Monte Carlo sampling from directed graphs with broad applicability. 

If the graph is constructed using distributions from the exponential family, and 
if the parent-child relationships preserve conjugacy, then the full conditional distri- 
butions arising in Gibbs sampling will have the same functional form as the orig- 
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Figure 11.12 The Gibbs sampling method requires samples 
to be drawn from the conditional distribution of a variable condi- 
tioned on the remaining variables. For graphical models, this 
conditional distribution is a function only of the states of the 
nodes in the Markov blanket. For an undirected graph this com- 
prises the set of neighbours, as shown on the left, while for a 
directed graph the Markov blanket comprises the parents, the 
children, and the co-parents, as shown on the right. 
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inal conditional distributions (conditioned on the parents) defining each node, and 
so standard sampling techniques can be employed. In general, the full conditional 
distributions will be of a complex form that does not permit the use of standard sam- 
pling algorithms. However, if these conditionals are log concave, then sampling can 
be done efficiently using adaptive rejection sampling (assuming the corresponding 
variable is a scalar). 

If, at each stage of the Gibbs sampling algorithm, instead of drawing a sample 
from the corresponding conditional distribution, we make a point estimate of the 
variable given by the maximum of the conditional distribution, then we obtain the 
iterated conditional modes (ICM) algorithm discussed in Section 8.3.3. Thus ICM 
can be seen as a greedy approximation to Gibbs sampling. 

Because the basic Gibbs sampling technique considers one variable at a time, 
there are strong dependencies between successive samples. At the opposite extreme, 
if we could draw samples directly from the joint distribution (an operation that we 
are supposing is intractable), then successive samples would be independent. We can 
hope to improve on the simple Gibbs sampler by adopting an intermediate strategy in 
which we sample successively from groups of variables rather than individual vari- 
ables. This is achieved in the blocking Gibbs sampling algorithm by choosing blocks 
of variables, not necessarily disjoint, and then sampling jointly from the variables in 
each block in turn, conditioned on the remaining variables (Jensen et al., 1995). 


Slice Sampling 


We have seen that one of the difficulties with the Metropolis algorithm is the sensi- 
tivity to step size. If this is too small, the result is slow decorrelation due to random 
walk behaviour, whereas if it is too large the result is inefficiency due to a high rejec- 
tion rate. The technique of slice sampling (Neal, 2003) provides an adaptive step size 
that is automatically adjusted to match the characteristics of the distribution. Again 
it requires that we are able to evaluate the unnormalized distribution p(z). 

Consider first the univariate case. Slice sampling involves augmenting z with 
an additional variable u and then drawing samples from the joint (z, u) space. We 
shall see another example of this approach when we discuss hybrid Monte Carlo in 
Section 11.5. The goal is to sample uniformly from the area under the distribution 
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Figure 11.13 Illustration of slice sampling. (a) For a given value z™, a value of u is chosen uniformly in 
the region 0 < u < p(z), which then defines a ‘slice’ through the distribution, shown by the solid horizontal 
lines. (b) Because it is infeasible to sample directly from a slice, a new sample of z is drawn from a region 
Zmin < Z < Zmax, which contains the previous value z0. 


given by 
1/Zp if0 < u< plz) 


11.51 
0 otherwise ( ) 


D(z, u) — { 


where Zp = f p(z) dz. The marginal distribution over z is given by 


p(z) ~ 
[ren du = f a du = aa = p(z) (11.52) 
0 P P 


and so we can sample from p(z) by sampling from p(z, u) and then ignoring the u 
values. This can be achieved by alternately sampling z and u. Given the value of z 
we evaluate p(z) and then sample u uniformly in the range 0 < u < p(z), which is 
straightforward. Then we fix u and sample z uniformly from the ‘slice’ through the 
distribution defined by {z : p(z) > u}. This is illustrated in Figure 11.13(a). 

In practice, it can be difficult to sample directly from a slice through the distribu- 
tion and so instead we define a sampling scheme that leaves the uniform distribution 
under p(z, u) invariant, which can be achieved by ensuring that detailed balance is 
satisfied. Suppose the current value of z is denoted 2‘) and that we have obtained 
a corresponding sample u. The next value of z is obtained by considering a region 
Zmin < Z < Zmax that contains z‘7). It is in the choice of this region that the adap- 
tation to the characteristic length scales of the distribution takes place. We want the 
region to encompass as much of the slice as possible so as to allow large moves in z 
space while having as little as possible of this region lying outside the slice, because 
this makes the sampling less efficient. 

One approach to the choice of region involves starting with a region containing 
z(7) having some width w and then testing each of the end points to see if they lie 
within the slice. If either end point does not, then the region is extended in that 
direction by increments of value w until the end point lies outside the region. A 
candidate value z’ is then chosen uniformly from this region, and if it lies within the 
slice, then it forms z‘7+!). If it lies outside the slice, then the region is shrunk such 
that z’ forms an end point and such that the region still contains 2”). Then another 


548 


11. SAMPLING METHODS 


11.5. 


candidate point is drawn uniformly from this reduced region and so on, until a value 
of z is found that lies within the slice. 

Slice sampling can be applied to multivariate distributions by repeatedly sam- 
pling each variable in turn, in the manner of Gibbs sampling. This requires that 
we are able to compute, for each component z;, a function that is proportional to 


p(z;|Z\;). 


The Hybrid Monte Carlo Algorithm 


As we have already noted, one of the major limitations of the Metropolis algorithm 
is that it can exhibit random walk behaviour whereby the distance traversed through 
the state space grows only as the square root of the number of steps. The problem 
cannot be resolved simply by taking bigger steps as this leads to a high rejection rate. 
In this section, we introduce a more sophisticated class of transitions based on an 
analogy with physical systems and that has the property of being able to make large 
changes to the system state while keeping the rejection probability small. It is ap- 
plicable to distributions over continuous variables for which we can readily evaluate 
the gradient of the log probability with respect to the state variables. We will discuss 
the dynamical systems framework in Section 11.5.1, and then in Section 11.5.2 we 
explain how this may be combined with the Metropolis algorithm to yield the pow- 
erful hybrid Monte Carlo algorithm. A background in physics is not required as this 
section is self-contained and the key results are all derived from first principles. 


11.5.1 Dynamical systems 


The dynamical approach to stochastic sampling has its origins in algorithms for 
simulating the behaviour of physical systems evolving under Hamiltonian dynam- 
ics. In a Markov chain Monte Carlo simulation, the goal is to sample from a given 
probability distribution p(z). The framework of Hamiltonian dynamics is exploited 
by casting the probabilistic simulation in the form of a Hamiltonian system. In order 
to remain in keeping with the literature in this area, we make use of the relevant 
dynamical systems terminology where appropriate, which will be defined as we go 
along. 

The dynamics that we consider corresponds to the evolution of the state variable 
z = {zi} under continuous time, which we denote by 7. Classical dynamics is de- 
scribed by Newton’s second law of motion in which the acceleration of an object is 
proportional to the applied force, corresponding to a second-order differential equa- 
tion over time. We can decompose a second-order equation into two coupled first- 
order equations by introducing intermediate momentum variables r, corresponding 
to the rate of change of the state variables z, having components 


= dz; 


Ti = 
dr 


(11.53) 


where the z; can be regarded as position variables in this dynamics perspective. Thus 
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for each position variable there is a corresponding momentum variable, and the joint 
space of position and momentum variables is called phase space. 
Without loss of generality, we can write the probability distribution p(z) in the 


form 


p(z) = exp (-E(2)) 


A (11.54) 


where E(z) is interpreted as the potential energy of the system when in state z. The 
system acceleration is the rate of change of momentum and is given by the applied 
force, which itself is the negative gradient of the potential energy 


dr; 
dr 


B OE(z) 
s= (11.55) 


It is convenient to reformulate this dynamical system using the Hamiltonian 
framework. To do this, we first define the kinetic energy by 


1 1 
K(r) = zll” = TD 


(11.56) 


a 


The total energy of the system is then the sum of its potential and kinetic energies 


H(z,r) = E(z)+ K(r) 


(11.57) 


where H is the Hamiltonian function. Using (11.53), (11.55), (11.56), and (11.57), 
we can now express the dynamics of the system in terms of the Hamiltonian equa- 


Exercise 11.15 tions given by 


dz; 2 OH 
ap (11.58) 
dr; OH 
GS ae (11.59) 


William Hamilton 
1805-1865 


William Rowan Hamilton was an 
Irish mathematician and physicist, 
and child prodigy, who was ap- 
pointed Professor of Astronomy at 
Trinity College, Dublin, in 1827, be- 
fore he had even graduated. One 
of Hamilton’s most important contributions was a new 
formulation of dynamics, which played a significant 
role in the later development of quantum mechanics. 


His other great achievement was the development of 
quaternions, which generalize the concept of complex 
numbers by introducing three distinct square roots of 
minus one, which satisfy i? = j? = k? = ijk = —1. 
It is said that these equations occurred to him while 
walking along the Royal Canal in Dublin with his wife, 
on 16 October 1843, and he promptly carved the 
equations into the side of Broome bridge. Although 
there is no longer any evidence of the carving, there is 
now a stone plaque on the bridge commemorating the 
discovery and displaying the quaternion equations. 
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During the evolution of this dynamical system, the value of the Hamiltonian H is 
constant, as is easily seen by differentiation 


dH _ 5y OH dz; OH dr; 
dr : Oz; dr Or; dr 


OHOH OHOH 
= Ne { pe oe } =0. (11.60) 


i 


A second important property of Hamiltonian dynamical systems, known as Li- 
ouville’s Theorem, is that they preserve volume in phase space. In other words, if 
we consider a region within the space of variables (z, r), then as this region evolves 
under the equations of Hamiltonian dynamics, its shape may change but its volume 
will not. This can be seen by noting that the flow field (rate of change of location in 


phase space) is given by 
dz dr 
V = | —, — 11.61 
( dr’ z) ( ) 
and that the divergence of this field vanishes 


: = 0 dz; (6) dr; 
a vfs dr Or; | 


ð ðH ð ðH 
= D A T =} 0. (11.62) 


Now consider the joint distribution over phase space whose total energy is the 
Hamiltonian, i.e., the distribution given by 


p(z,r) = dea, (11.63) 
ZH 
Using the two results of conservation of volume and conservation of H, it follows 
that the Hamiltonian dynamics will leave p(z,r) invariant. This can be seen by 
considering a small region of phase space over which H is approximately constant. 
If we follow the evolution of the Hamiltonian equations for a finite time, then the 
volume of this region will remain unchanged as will the value of H in this region, and 
hence the probability density, which is a function only of H, will also be unchanged. 
Although H is invariant, the values of z and r will vary, and so by integrating 
the Hamiltonian dynamics over a finite time duration it becomes possible to make 
large changes to z in a systematic way that avoids random walk behaviour. 
Evolution under the Hamiltonian dynamics will not, however, sample ergodi- 
cally from p(z, r) because the value of H is constant. In order to arrive at an ergodic 
sampling scheme, we can introduce additional moves in phase space that change 
the value of H while also leaving the distribution p(z,r) invariant. The simplest 
way to achieve this is to replace the value of r with one drawn from its distribution 
conditioned on z. This can be regarded as a Gibbs sampling step, and hence from 


Exercise 11.16 


11.5. The Hybrid Monte Carlo Algorithm 551 


Section 11.3 we see that this also leaves the desired distribution invariant. Noting 
that z and r are independent in the distribution p(z, r), we see that the conditional 
distribution p(r|z) is a Gaussian from which it is straightforward to sample. 

In a practical application of this approach, we have to address the problem of 
performing a numerical integration of the Hamiltonian equations. This will neces- 
sarily introduce numerical errors and so we should devise a scheme that minimizes 
the impact of such errors. In fact, it turns out that integration schemes can be devised 
for which Liouville’s theorem still holds exactly. This property will be important in 
the hybrid Monte Carlo algorithm, which is discussed in Section 11.5.2. One scheme 
for achieving this is called the leapfrog discretization and involves alternately updat- 
ing discrete-time approximations Z and T to the position and momentum variables 
using 


(r+ e/2) = Rl- SSe el) (11.64) 
Zi(t+e) = lr) + Tlr + €/2) (11.65) 
Tre) = TF + e€/2) OE ar + €)). (11.66) 


We see that this takes the form of a half-step update of the momentum variables with 
step size €/2, followed by a full-step update of the position variables with step size €, 
followed by a second half-step update of the momentum variables. If several leapfrog 
steps are applied in succession, it can be seen that half-step updates to the momentum 
variables can be combined into full-step updates with step size e. The successive 
updates to position and momentum variables then leapfrog over each other. In order 
to advance the dynamics by a time interval r, we need to take 7/e steps. The error 
involved in the discretized approximation to the continuous time dynamics will go to 
zero, assuming a smooth function F(z), in the limit e — 0. However, for a nonzero 
€ as used in practice, some residual error will remain. We shall see in Section 11.5.2 
how the effects of such errors can be eliminated in the hybrid Monte Carlo algorithm. 

In summary then, the Hamiltonian dynamical approach involves alternating be- 
tween a Series of leapfrog updates and a resampling of the momentum variables from 
their marginal distribution. 

Note that the Hamiltonian dynamics method, unlike the basic Metropolis algo- 
rithm, is able to make use of information about the gradient of the log probability 
distribution as well as about the distribution itself. An analogous situation is familiar 
from the domain of function optimization. In most cases where gradient informa- 
tion is available, it is highly advantageous to make use of it. Informally, this follows 
from the fact that in a space of dimension D, the additional computational cost of 
evaluating a gradient compared with evaluating the function itself will typically be a 
fixed factor independent of D, whereas the D-dimensional gradient vector conveys 
D pieces of information compared with the one piece of information given by the 
function itself. 
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11.5.2 Hybrid Monte Carlo 


As we discussed in the previous section, for a nonzero step size e€, the discretiza- 
tion of the leapfrog algorithm will introduce errors into the integration of the Hamil- 
tonian dynamical equations. Hybrid Monte Carlo (Duane et al., 1987; Neal, 1996) 
combines Hamiltonian dynamics with the Metropolis algorithm and thereby removes 
any bias associated with the discretization. 

Specifically, the algorithm uses a Markov chain consisting of alternate stochastic 
updates of the momentum variable r and Hamiltonian dynamical updates using the 
leapfrog algorithm. After each application of the leapfrog algorithm, the resulting 
candidate state is accepted or rejected according to the Metropolis criterion based 
on the value of the Hamiltonian H. Thus if (z,r) is the initial state and (z*,r*) 
is the state after the leapfrog integration, then this candidate state is accepted with 
probability 

min (1,exp{H(z,r) — H(z*,r*)}). (11.67) 


If the leapfrog integration were to simulate the Hamiltonian dynamics perfectly, 
then every such candidate step would automatically be accepted because the value 
of H would be unchanged. Due to numerical errors, the value of H may sometimes 
decrease, and we would like the Metropolis criterion to remove any bias due to this 
effect and ensure that the resulting samples are indeed drawn from the required dis- 
tribution. In order for this to be the case, we need to ensure that the update equations 
corresponding to the leapfrog integration satisfy detailed balance (11.40). This is 
easily achieved by modifying the leapfrog scheme as follows. 

Before the start of each leapfrog integration sequence, we choose at random, 
with equal probability, whether to integrate forwards in time (using step size €) or 
backwards in time (using step size —e). We first note that the leapfrog integration 
scheme (11.64), (11.65), and (11.66) is time-reversible, so that integration for L steps 
using step size —e will exactly undo the effect of integration for L steps using step 
size €. Next we show that the leapfrog integration preserves phase-space volume 
exactly. This follows from the fact that each step in the leapfrog scheme updates 
either a z; variable or an r; variable by an amount that is a function only of the other 
variable. As shown in Figure 11.14, this has the effect of shearing a region of phase 
space while not altering its volume. 

Finally, we use these results to show that detailed balance holds. Consider a 
small region R of phase space that, under a sequence of L leapfrog iterations of 
step size €, maps to a region R’. Using conservation of volume under the leapfrog 
iteration, we see that if R has volume ôV then so too will R’. If we choose an initial 
point from the distribution (11.63) and then update it using L leapfrog interactions, 
the probability of the transition going from R to R’ is given by 


= exp(-H(R))V'5 min {1,exp(—H(R) + H(R’))}. (11.68) 
H 


where the factor of 1/2 arises from the probability of choosing to integrate with a 
positive step size rather than a negative one. Similarly, the probability of starting in 


11.5. The Hybrid Monte Carlo Algorithm 553 


1 
Zi Zi 


Figure 11.14 Each step of the leapfrog algorithm (11.64)-(11.66) modifies either a position variable z; or a 
momentum variable r;. Because the change to one variable is a function only of the other, any region in phase 
space will be sheared without change of volume. 


Exercise 11.17 


region R’ and integrating backwards in time to end up in region R is given b 
g g g p g g y 


= exp(-H(R'))iV 5 min {1,exp(—H(R’) + H(R))}. (11.69) 
H 

It is easily seen that the two probabilities (11.68) and (11.69) are equal, and hence 
detailed balance holds. Note that this proof ignores any overlap between the regions 
R and R’ but is easily generalized to allow for such overlap. 

It is not difficult to construct examples for which the leapfrog algorithm returns 
to its starting position after a finite number of iterations. In such cases, the random 
replacement of the momentum values before each leapfrog integration will not be 
sufficient to ensure ergodicity because the position variables will never be updated. 
Such phenomena are easily avoided by choosing the magnitude of the step size at 
random from some small interval, before each leapfrog integration. 

We can gain some insight into the behaviour of the hybrid Monte Carlo algo- 
rithm by considering its application to a multivariate Gaussian. For convenience, 
consider a Gaussian distribution p(z) with independent components, for which the 
Hamiltonian is given by 


| 1 
H(z,r) = Se es (11.70) 


Our conclusions will be equally valid for a Gaussian distribution having correlated 
components because the hybrid Monte Carlo algorithm exhibits rotational isotropy. 
During the leapfrog integration, each pair of phase-space variables z;, r; evolves in- 
dependently. However, the acceptance or rejection of the candidate point is based 
on the value of H, which depends on the values of all of the variables. Thus, a 
significant integration error in any one of the variables could lead to a high prob- 
ability of rejection. In order that the discrete leapfrog integration be a reasonably 
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11.6. 


good approximation to the true continuous-time dynamics, it is necessary for the 
leapfrog integration scale € to be smaller than the shortest length-scale over which 
the potential is varying significantly. This is governed by the smallest value of oi, 
which we denote by domin. Recall that the goal of the leapfrog integration in hybrid 
Monte Carlo is to move a substantial distance through phase space to a new state 
that is relatively independent of the initial state and still achieve a high probability of 
acceptance. In order to achieve this, the leapfrog integration must be continued for a 
number of iterations of order Omax /Omin- 

By contrast, consider the behaviour of a simple Metropolis algorithm with an 
isotropic Gaussian proposal distribution of variance s”, considered earlier. In order 
to avoid high rejection rates, the value of s must be of order omin. The exploration of 
state space then proceeds by a random walk and takes of order (Omax/Omin)* Steps 
to arrive at a roughly independent state. 


Estimating the Partition Function 


As we have seen, most of the sampling algorithms considered in this chapter re- 
quire only the functional form of the probability distribution up to a multiplicative 
constant. Thus if we write 


pp(2) = — exp(—E(@)) a171) 
ZE 

then the value of the normalization constant Zg, also known as the partition func- 
tion, is not needed in order to draw samples from p(z). However, knowledge of the 
value of Zz can be useful for Bayesian model comparison since it represents the 
model evidence (i.e., the probability of the observed data given the model), and so 
it is of interest to consider how its value might be obtained. We assume that direct 
evaluation by summing, or integrating, the function exp(— E(z)) over the state space 
of z is intractable. 

For model comparison, it is actually the ratio of the partition functions for two 
models that is required. Multiplication of this ratio by the ratio of prior probabilities 
gives the ratio of posterior probabilities, which can then be used for model selection 
or model averaging. 

One way to estimate a ratio of partition functions is to use importance sampling 
from a distribution with energy function G(z) 


Zg diz exp(—E(z)) 
Za diz exp(—G(z)) 
>, exp(—E(z) + G(z)) exp(—G(z)) 
di, exp(—G(z)) 
= Ege@lexp(-E + G)] 


S exp(-E(z®) + G2) (11.72) 
l 
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where {z!)} are samples drawn from the distribution defined by pg(z). If the dis- 
tribution pq is one for which the partition function can be evaluated analytically, for 
example a Gaussian, then the absolute value of Zp can be obtained. 

This approach will only yield accurate results if the importance sampling distri- 
bution pg is closely matched to the distribution pg, so that the ratio pz /pg does not 
have wide variations. In practice, suitable analytically specified importance sampling 
distributions cannot readily be found for the kinds of complex models considered in 
this book. 

An alternative approach is therefore to use the samples obtained from a Markov 
chain to define the importance-sampling distribution. If the transition probability for 
the Markov chain is given by T(z, z’), and the sample set is given by 2), ...,24), 
then the sampling distribution can be written as 


1 
a= Ck T(2 2 (11.73) 


which can be used directly in (11.72). 
Methods for estimating the ratio of two partition functions require for their suc- 
cess that the two corresponding distributions be reasonably closely matched. This is 
especially problematic if we wish to find the absolute value of the partition function 
for a complex distribution because it is only for relatively simple distributions that 
the partition function can be evaluated directly, and so attempting to estimate the 
ratio of partition functions directly is unlikely to be successful. This problem can be 
tackled using a technique known as chaining (Neal, 1993; Barber and Bishop, 1997), 
which involves introducing a succession of intermediate distributions p2,..., Pag—1 
that interpolate between a simple distribution p;(z) for which we can evaluate the 
normalization coefficient Z, and the desired complex distribution pjy(z). We then 
have 
Zm _ 4243; Zm 
4,0 Z% Zma 


in which the intermediate ratios can be determined using Monte Carlo methods as 
discussed above. One way to construct such a sequence of intermediate systems 
is to use an energy function containing a continuous parameter 0 < œ < 1 that 
interpolates between the two distributions 


(11.74) 


Ealz) = (1 — a) E (z) + aEm (z). (11.75) 


If the intermediate ratios in (11.74) are to be found using Monte Carlo, it may be 
more efficient to use a single Markov chain run than to restart the Markov chain for 
each ratio. In this case, the Markov chain is run initially for the system pı and then 
after some suitable number of steps moves on to the next distribution in the sequence. 
Note, however, that the system must remain close to the equilibrium distribution at 
each stage. 
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Exercises 
11.1 


11.2 


11.10 


(x) FY Show that the finite sample estimator F defined by (11.2) has mean 
equal to E[f] and variance given by (11.3). 


(x) Suppose that z is a random variable with uniform distribution over (0,1) and 
that we transform z using y = h~'(z) where h(y) is given by (11.6). Show that y 
has the distribution p(y). 


(x) Given a random variable z that is uniformly distributed over (0, 1), find a trans- 
formation y = f(z) such that y has a Cauchy distribution given by (11.8). 


(xx) Suppose that zı and z are uniformly distributed over the unit circle, as 
shown in Figure 11.3, and that we make the change of variables given by (11.10) 
and (11.11). Show that (y1, y2) will be distributed according to (11.12). 


(x) www | Let z be a D-dimensional random variable having a Gaussian distribu- 
tion with zero mean and unit covariance matrix, and suppose that the positive definite 
symmetric matrix © has the Cholesky decomposition © = LL" where L is a lower- 
triangular matrix (i.e., one with zeros above the leading diagonal). Show that the 
variable y = u + Lz has a Gaussian distribution with mean ps and covariance X. 
This provides a technique for generating samples from a general multivariate Gaus- 
sian using samples from a univariate Gaussian having zero mean and unit variance. 


œ» FY In this exercise, we show more carefully that rejection sampling does 
indeed draw samples from the desired distribution p(z). Suppose the proposal dis- 
tribution is g(z) and show that the probability of a sample value z being accepted is 
given by p(z)/kq(z) where p is any unnormalized distribution that is proportional to 
p(z), and the constant k is set to the smallest value that ensures kq(z) > p(z) for all 
values of z. Note that the probability of drawing a value z is given by the probability 
of drawing that value from q(z) times the probability of accepting that value given 
that it has been drawn. Make use of this, along with the sum and product rules of 
probability, to write down the normalized form for the distribution over z, and show 
that it equals p(z). 


(x) Suppose that z has a uniform distribution over the interval [0, 1]. Show that the 
variable y = btan z + c has a Cauchy distribution given by (11.16). 


(xx) Determine expressions for the coefficients k; in the envelope distribution 
(11.17) for adaptive rejection sampling using the requirements of continuity and nor- 
malization. 


(xx) By making use of the technique discussed in Section 11.1.1 for sampling 
from a single exponential distribution, devise an algorithm for sampling from the 
piecewise exponential distribution defined by (11.17). 


(x) Show that the simple random walk over the integers defined by (11.34), (11.35), 
and (11.36) has the property that E[(z‘”)?] = E[(z°"-))?] + 1/2 and hence by 
induction that E[(z‘7))?] = 7/2. 


Figure 11.15 


11.11 


11.12 


11.13 


11.14 


11.15 


11.16 


Figure 11.16 
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A probability distribution over two variables zı 29 
and z2 that is uniform over the shaded regions 
and that is zero everywhere else. 


Z1 


œ» EA Show that the Gibbs sampling algorithm, discussed in Section 11.3, 
satisfies detailed balance as defined by (11.40). 


(x) Consider the distribution shown in Figure 11.15. Discuss whether the standard 
Gibbs sampling procedure for this distribution is ergodic, and therefore whether it 
would sample correctly from this distribution 


(x x) Consider the simple 3-node graph shown in Figure 11.16 in which the observed 
node x is given by a Gaussian distribution N(x|,7~') with mean y and precision 
T. Suppose that the marginal distributions over the mean and precision are given 
by N (u| uo, So) and Gam(r]a, b), where Gam(-|-,-) denotes a gamma distribution. 
Write down expressions for the conditional distributions p(u|x, 7) and p(T|x, p) that 
would be required in order to apply Gibbs sampling to the posterior distribution 


plu, 7 |x). 
(x) Verify that the over-relaxation update (11.50), in which z; has mean u; and 


variance o;, and where v has zero mean and unit variance, gives a value 2 with 
mean u; and variance o?. 


œ UY Using (11.56) and (11.57), show that the Hamiltonian equation (11.58) 
is equivalent to (11.53). Similarly, using (11.57) show that (11.59) is equivalent to 
(11.55). 


(x) By making use of (11.56), (11.57), and (11.63), show that the conditional dis- 
tribution p(r|z) is a Gaussian. 


A graph involving an observed Gaussian variable x with H T 
prior distributions over its mean y and precision 7. 
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11.17 («) [QQ Verify that the two probabilities (11.68) and (11.69) are equal, and hence 
that detailed balance holds for the hybrid Monte Carlo algorithm. 


Appendix A 


12 


Continuous | 


In Chapter 9, we discussed probabilistic models having discrete latent variables, such 
as the mixture of Gaussians. We now explore models in which some, or all, of the 
latent variables are continuous. An important motivation for such models is that 
many data sets have the property that the data points all lie close to a manifold of 
much lower dimensionality than that of the original data space. To see why this 
might arise, consider an artificial data set constructed by taking one of the off-line 
digits, represented by a 64 x 64 pixel grey-level image, and embedding it in a larger 
image of size 100 x 100 by padding with pixels having the value zero (corresponding 
to white pixels) in which the location and orientation of the digit is varied at random, 
as illustrated in Figure 12.1. Each of the resulting images is represented by a point in 
the 100 x 100 = 10, 000-dimensional data space. However, across a data set of such 
images, there are only three degrees of freedom of variability, corresponding to the 
vertical and horizontal translations and the rotations. The data points will therefore 
live on a subspace of the data space whose intrinsic dimensionality is three. Note 
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Figure 12.1 A synthetic data set obtained by taking one of the off-line digit images and creating multi- 
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Section 12.1 


ple copies in each of which the digit has undergone a random displacement and rotation 
within some larger image field. The resulting images each have 100 x 100 = 10,000 
pixels. 


that the manifold will be nonlinear because, for instance, if we translate the digit 
past a particular pixel, that pixel value will go from zero (white) to one (black) and 
back to zero again, which is clearly a nonlinear function of the digit position. In 
this example, the translation and rotation parameters are latent variables because we 
observe only the image vectors and are not told which values of the translation or 
rotation variables were used to create them. 

For real digit image data, there will be a further degree of freedom arising from 
scaling. Moreover there will be multiple additional degrees of freedom associated 
with more complex deformations due to the variability in an individual’s writing 
as well as the differences in writing styles between individuals. Nevertheless, the 
number of such degrees of freedom will be small compared to the dimensionality of 
the data set. 

Another example is provided by the oil flow data set, in which (for a given ge- 
ometrical configuration of the gas, water, and oil phases) there are only two degrees 
of freedom of variability corresponding to the fraction of oil in the pipe and the frac- 
tion of water (the fraction of gas then being determined). Although the data space 
comprises 12 measurements, a data set of points will lie close to a two-dimensional 
manifold embedded within this space. In this case, the manifold comprises several 
distinct segments corresponding to different flow regimes, each such segment being 
a (noisy) continuous two-dimensional manifold. If our goal is data compression, or 
density modelling, then there can be benefits in exploiting this manifold structure. 

In practice, the data points will not be confined precisely to a smooth low- 
dimensional manifold, and we can interpret the departures of data points from the 
manifold as ‘noise’. This leads naturally to a generative view of such models in 
which we first select a point within the manifold according to some latent variable 
distribution and then generate an observed data point by adding noise, drawn from 
some conditional distribution of the data variables given the latent variables. 

The simplest continuous latent variable model assumes Gaussian distributions 
for both the latent and observed variables and makes use of a linear-Gaussian de- 
pendence of the observed variables on the state of the latent variables. This leads 
to a probabilistic formulation of the well-known technique of principal component 
analysis (PCA), as well as to a related model called factor analysis. 

In this chapter w will begin with a standard, nonprobabilistic treatment of PCA, 
and then we show how PCA arises naturally as the maximum likelihood solution to 
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Principal component analysis seeks a space uj 
of lower dimensionality, known as the princi- PA 
pal subspace and denoted by the magenta 
line, such that the orthogonal projection of 
the data points (red dots} onto this subspace 
maximizes the variance of the projected points 
(green dots). An alternative definition of PCA 
is based on minimizing the sum-of-squares 
of the projection errors, indicated by the blue 
lines. 


T2 


Tı 


a particular form of linear-Gaussian latent variable model. This probabilistic refor- 
mulation brings many advantages, such as the use of EM for parameter estimation, 
principled extensions to mixtures of PCA models, and Bayesian formulations that 
allow the number of principal components to be determined automatically from the 
data. Finally, we discuss briefly several generalizations of the latent variable concept 
that go beyond the linear-Gaussian assumption including non-Gaussian latent vari- 
ables, which leads to the framework of independent component analysis, as well as 
models having a nonlinear relationship between latent and observed variables. 


Principal Component Analysis 


Principal component analysis, or PCA, is a technique that is widely used for appli- 
cations such as dimensionality reduction, lossy data compression, feature extraction, 
and data visualization (Jolliffe, 2002). It is also known as the Karhunen-Loève trans- 
form. 

There are two commonly used definitions of PCA that give rise to the same 
algorithm. PCA can be defined as the orthogonal projection of the data onto a lower 
dimensional linear space, known as the principal subspace, such that the variance of 
the projected data is maximized (Hotelling, 1933). Equivalently, it can be defined as 
the linear projection that minimizes the average projection cost, defined as the mean 
squared distance between the data points and their projections (Pearson, 1901). The 
process of orthogonal projection is illustrated in Figure 12.2. We consider each of 
these definitions in turn. 


12.1.1 Maximum variance formulation 


Consider a data set of observations {Xn} where n = 1,..., N, and x, is a 
Euclidean variable with dimensionality D. Our goal is to project the data onto a 
space having dimensionality M < D while maximizing the variance of the projected 
data. For the moment, we shall assume that the value of M is given. Later in this 
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chapter, we shall consider techniques to determine an appropriate value of M from 
the data. 

To begin with, consider the projection onto a one-dimensional space (M = 1). 
We can define the direction of this space using a D-dimensional vector u1, which 
for convenience (and without loss of generality) we shall choose to be a unit vector 
so that uj u; = 1 (note that we are only interested in the direction defined by uy, 
not in the magnitude of u; itself). Each data point x, is then projected onto a scalar 
value uj xn. The mean of the projected data is u? x where X is the sample set mean 
given by 


1 N 
X= Wx (12.1) 
n=l 


and the variance of the projected data is given by 


N 
=> fulx, — ulx}* = uTSu, (12.2) 


n=1 


where S is the data covariance matrix defined by 
1 & 
S=5 Dn =x)G — X)T. (12.3) 
n= 


We now maximize the projected variance uT Su, with respect to u,. Clearly, this has 
to be a constrained maximization to prevent |lu; || — oo. The appropriate constraint 
comes from the normalization condition ulu; = 1. To enforce this constraint, 
we introduce a Lagrange multiplier that we shall denote by 4;, and then make an 
unconstrained maximization of 


uj Su; + A; (1 — ufu). (12.4) 


By setting the derivative with respect to u, equal to zero, we see that this quantity 
will have a stationary point when 


Su; = Ayu, (12.5) 


which says that u; must be an eigenvector of S. If we left-multiply by uT and make 
use of uf u; = 1, we see that the variance is given by 


uj; Su; = A; (12.6) 


and so the variance will be a maximum when we set u, equal to the eigenvector 
having the largest eigenvalue A,. This eigenvector is known as the first principal 
component. 

We can define additional principal components in an incremental fashion by 
choosing each new direction to be that which maximizes the projected variance 
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amongst all possible directions orthogonal to those already considered. If we con- 
sider the general case of an M-dimensional projection space, the optimal linear pro- 
jection for which the variance of the projected data is maximized is now defined by 
the M eigenvectors uj, ..., um of the data covariance matrix S corresponding to the 
M largest eigenvalues \,,..., Am. This is easily shown using proof by induction. 

To summarize, principal component analysis involves evaluating the mean X 
and the covariance matrix S of the data set and then finding the M eigenvectors of S 
corresponding to the M largest eigenvalues. Algorithms for finding eigenvectors and 
eigenvalues, as well as additional theorems related to eigenvector decomposition, 
can be found in Golub and Van Loan (1996). Note that the computational cost of 
computing the full eigenvector decomposition for a matrix of size D x D is O(D*). 
If we plan to project our data onto the first M principal components, then we only 
need to find the first M eigenvalues and eigenvectors. This can be done with more 
efficient techniques, such as the power method (Golub and Van Loan, 1996), that 
scale like O(M D°), or alternatively we can make use of the EM algorithm. 


12.1.2 Minimum-error formulation 


We now discuss an alternative formulation of PCA based on projection error 
minimization. To do this, we introduce a complete orthonormal set of D-dimensional 
basis vectors {u;} where i = 1,..., D that satisfy 


uy uj = ĵij- (12.7) 


Because this basis is complete, each data point can be represented exactly by a linear 
combination of the basis vectors 


D 
x = > Oni; (12.8) 
i=1 


where the coefficients œn; will be different for different data points. This simply 
corresponds to a rotation of the coordinate system to a new system defined by the 
{u;}, and the original D components {2,1,...,2np} are replaced by an equivalent 
set {Qni,.--,;Qnp}. Taking the inner product with u;, and making use of the or- 
thonormality property, we obtain an; = x} uj, and so without loss of generality we 


can write 
D 


Xn = ` (xnu) u;. (12.9) 
i=l 
Our goal, however, is to approximate this data point using a representation in- 
volving a restricted number M < D of variables corresponding to a projection onto 
a lower-dimensional subspace. The M-dimensional linear subspace can be repre- 
sented, without loss of generality, by the first M of the basis vectors, and so we 
approximate each data point x,, by 


M D 
Xn =Y maui + os biu; (12.10) 
i=l 


i=M+1 


564 


12. CONTINUOUS LATENT VARIABLES 


where the {2,,;} depend on the particular data point, whereas the {b;} are constants 
that are the same for all data points. We are free to choose the {u;}, the {zni}, and 
the {b;} so as to minimize the distortion introduced by the reduction in dimensional- 
ity. As our distortion measure, we shall use the squared distance between the original 
data point x,, and its approximation x,,, averaged over the data set, so that our goal 
is to minimize 


1 N 
j= N len — Rall? (12.11) 


Consider first of all the minimization with respect to the quantities {zn;}. Sub- 
stituting for X,,, setting the derivative with respect to zn; to zero, and making use of 
the orthonormality conditions, we obtain 

Znj = ee (12.12) 


where j = 1,..., M. Similarly, setting the derivative of J with respect to b; to zero, 
and again making use of the orthonormality relations, gives 


bj =x" u; (12.13) 
where j = M +1,..., D. If we substitute for zn; and b;, and make use of the general 
expansion (12.9), we obtain 

D 
Xn — Fn = Š {(%_p —¥)T uy} uy (12.14) 
i=M+1 


from which we see that the displacement vector from Xn to X, lies in the space 
orthogonal to the principal subspace, because it is a linear combination of {u;} for 
i = M +1,...,D, as illustrated in Figure 12.2. This is to be expected because the 
projected points x,, must lie within the principal subspace, but we can move them 
freely within that subspace, and so the minimum error is given by the orthogonal 
projection. 

We therefore obtain an expression for the distortion measure J as a function 
purely of the {u,} in the form 


1 ee D 
I= DU S aw- xTui) = | Sul Su. (12.15) 
n=l i=M+1 i=M+1 


There remains the task of minimizing J with respect to the {u;}, which must 
be a constrained minimization otherwise we will obtain the vacuous result u; = 0. 
The constraints arise from the orthonormality conditions and, as we shall see, the 
solution will be expressed in terms of the eigenvector expansion of the covariance 
matrix. Before considering a formal solution, let us try to obtain some intuition about 
the result by considering the case of a two-dimensional data space D = 2 and a one- 
dimensional principal subspace M = 1. We have to choose a direction ux so as to 
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minimize J = us Sug, subject to the normalization constraint ud uz = 1. Using a 
Lagrange multiplier A» to enforce the constraint, we consider the minimization of 


J= u7 Suz + A2 (1 — uzu) ; (12.16) 


Setting the derivative with respect to ug to zero, we obtain Suz = A2uz so that uz 
is an eigenvector of S with eigenvalue A. Thus any eigenvector will define a sta- 
tionary point of the distortion measure. To find the value of J at the minimum, we 
back-substitute the solution for uz into the distortion measure to give J = Ag. We 
therefore obtain the minimum value of J by choosing ug to be the eigenvector corre- 
sponding to the smaller of the two eigenvalues. Thus we should choose the principal 
subspace to be aligned with the eigenvector having the larger eigenvalue. This result 
accords with our intuition that, in order to minimize the average squared projection 
distance, we should choose the principal component subspace to pass through the 
mean of the data points and to be aligned with the directions of maximum variance. 
For the case when the eigenvalues are equal, any choice of principal direction will 
give rise to the same value of J. 

The general solution to the minimization of J for arbitrary D and arbitrary M < 
D is obtained by choosing the {u;} to be eigenvectors of the covariance matrix given 
by 


Su; = Aiu; (12.17) 
where i = 1,..., D, and as usual the eigenvectors {u;} are chosen to be orthonor- 
mal. The corresponding value of the distortion measure is then given by 

D 
J= Yo OA (12.18) 
i=M41 


which is simply the sum of the eigenvalues of those eigenvectors that are orthogonal 
to the principal subspace. We therefore obtain the minimum value of J by selecting 
these eigenvectors to be those having the D — M smallest eigenvalues, and hence 
the eigenvectors defining the principal subspace are those corresponding to the M 
largest eigenvalues. 

Although we have considered M < D, the PCA analysis still holds if M = 
D, in which case there is no dimensionality reduction but simply a rotation of the 
coordinate axes to align with principal components. 

Finally, it is worth noting that there exists a closely related linear dimensionality 
reduction technique called canonical correlation analysis, or CCA (Hotelling, 1936; 
Bach and Jordan, 2002). Whereas PCA works with a single random variable, CCA 
considers two (or more) variables and tries to find a corresponding pair of linear 
subspaces that have high cross-correlation, so that each component within one of the 
subspaces is correlated with a single component from the other subspace. Its solution 
can be expressed in terms of a generalized eigenvector problem. 


12.1.3 Applications of PCA 


We can illustrate the use of PCA for data compression by considering the off- 
line digits data set. Because each eigenvector of the covariance matrix is a vector 
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Figure 12.3 The mean vector x along with the first four PCA eigenvectors uj, ....u, for the off-line 
digits data set, together with the corresponding eigenvalues. 


in the original D-dimensional space, we can represent the eigenvectors as images of 
the same size as the data points. The first five eigenvectors, along with the corre- 
sponding eigenvalues, are shown in Figure 12.3. A plot of the complete spectrum of 
eigenvalues, sorted into decreasing order, is shown in Figure 12.4(a). The distortion 
measure J associated with choosing a particular value of M is given by the sum 
of the eigenvalues from Af + 1 up to D and is plotted for different values of M in 
Figure 12.4(b). 

If we substitute (12.12) and (12.13) into (12.10), we can write the PCA approx- 
imation to a data vector x,, in the form 


M D 
Xn = So (xtuy)uy + J. (x? u,)u, (12.19) 
t=1 i=M +1 
M 
= X+) (xiu; - X"u;) u; (12.20) 
im] 
x 10° x 10° 
3 
3 J 
Ay 2 
2 
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Figure 12.4 (a) Plot of the eigenvalue spectrum for the off-line digits data set. (b) Plot of the sum of the 
discarded eigenvalues, which represents the sum-of-squares distortion J introduced by projecting the data onto 
a principal component subspace of dimensionality M. 
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Figure 12.5 An original example from the off-line digits data set together with its PCA reconstructions 
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obtained by retaining .W principal components for various values of M. As M increases 
the reconstruction becomes more accurate and would become perfect when M = D = 
28 x 28 = 784. 


where we have made use of the relation 


(x' uj) ui (12.21) 
1 


D 
x= 


t 
which follows from the completeness of the {u;}. This represents a compression 
of the data set, because for each data point we have replaced the D-dimensional 
vector X,, with an M-dimensional vector having components (xt u; — X Tui). The 
smaller the value of M, the greater the degree of compression. Examples of PCA 
reconstructions of data points for the digits data set are shown in Figure 12.5. 

Another application of principal component analysis is to data pre-processing. 
In this case, the goal is not dimensionality reduction but rather the transformation of 
a data set in order to standardize certain of its properties. This can be important in 
allowing subsequent pattern recognition algorithms to be applied successfully to the 
data set. Typically, itis done when the original variables are measured in various dif- 
ferent units or have significantly different variability. For instance in the Old Faithful 
data set, the time between eruptions is typically an order of magnitude greater than 
the duration of an eruption. When we applied the A’-means algorithm to this data 
set, we first made a separate linear re-scaling of the individual variables such that 


each variable had zero mean and unit variance. This is known as standardizing the 
data, and the covariance matrix for the standardized data has components 


N — 
= (Tni — Ti) ) (£n; — Tj) 
=1 


oj 


Pij (12.22) 


where g; is the variance of «;. This is known as the correlation matrix of the original 
data and has the property that if two components x; and x; of the data are perfectly 
correlated, then p,; = 1, and if they are uncorrelated, then p;; = 0. 

However, using PCA we can make a more substantial normalization of the data 
to give it zero mean and unit covariance, so that different variables become decorre- 
lated, To do this, we first write the eigenvector equation (12,17) in the form 


SU = UL (12.23) 
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Illustration of the effects of linear pre-processing applied to the Old Faithful data set. The plot on 


the left shows the original data. The centre plot shows the result of standardizing the individual variables to zero 
mean and unit variance. Also shown are the principal axes of this normalized data set, plotted over the range 


H. The plot on the right shows the result of whitening of the data to give it zero mean and unit covariance. 
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where L is a D x D diagonal matrix with elements \;, and U is a D x D orthog- 
onal matrix with columns given by u;. Then we define, for each data point xp, a 
transformed value given by 


Yn = LU" (xn — X) (12.24) 


where X is the sample mean defined by (12.1). Clearly, the set {yn } has zero mean, 
and its covariance is given by the identity matrix because 


N N 

1 1 

A Na = NW SOL PUT x, ~ X) (xn - X) TUL"? 
n=1 n=l 


L 'ZUTSUL = Lu O? =I. (12.25) 


This operation is known as whitening or sphereing the data and is illustrated for the 
Old Faithful data set in Figure 12.6. 

It is interesting to compare PCA with the Fisher linear discriminant which was 
discussed in Section 4.1.4. Both methods can be viewed as techniques for linear 
dimensionality reduction. However, PCA is unsupervised and depends only on the 
values x,, whereas Fisher linear discriminant also uses class-label information. This 
difference is highlighted by the example in Figure 12.7. 

Another common application of principal component analysis is to data visual- 
ization. Here each data point is projected onto a two-dimensional (M = 2) principal 
subspace, so that a data point Xn is plotted at Cartesian coordinates given by x! uy 
and x} us, where u; and up are the eigenvectors corresponding to the largest and 
second largest eigenvalues. An example of such a plot, for the oil flow data set, is 
shown in Figure 12.8. 


Figure 12.7 


Figure 12.8 
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A comparison of principal compo- 
nent analysis with Fisher's linear 
discriminant for linear dimension- 
ality reduction. Here the data in 
two dimensions, belonging to two 
classes shown in red and blue, is 
to be projected onto a single di- 
mension. PCA chooses the direc- 
tion of maximum variance, shown 
by the magenta curve, which leads 
to strong class overlap, whereas 
the Fisher linear discriminant takes 
account of the class labels and 
leads to a projection onto the green 
curve giving much better class 
separation. 


Visualization of the oil flow data set obtained 
by projecting the data onto the first two prin- 
cipal components. The red, blue, and green 
points correspond to the ‘laminar’, ‘homo- 
geneous’, and ‘annular’ flow configurations 
respectively. 


12.1.4 PCA for high-dimensional data 


In some applications of principal component analysis, the number of data points 
is smaller than the dimensionality of the data space. For example, we might want to 
apply PCA to a data set of a few hundred images, each of which corresponds to a 
vector in a space of potentially several million dimensions (corresponding to three 
colour values for each of the pixels in the image). Note that in a D-dimensional space 
a set of N points, where N < D, defines a linear subspace whose dimensionality 
is at most N — 1, and so there is little point in applying PCA for values of M 
that are greater than NV — 1. Indeed, if we perform PCA we will find that at least 
D — N + 1 of the eigenvalues are zero, corresponding to eigenvectors along whose 
directions the data set has zero variance. Furthermore, typical algorithms for finding 
the eigenvectors of a D x D matrix have a computational cost that scales like O( D°), 
and so for applications such as the image example, a direct application of PCA will 
be computationally infeasible. 

We can resolve this problem as follows. First, let us define X to be the (N x D)- 
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12.2. 


dimensional centred data matrix, whose nt? row is given by (x,, — X)”. The covari- 
N 


ance matrix (12.3) can then be written as S = NTIXTX, and the corresponding 
eigenvector equation becomes 


1 
ye Xu = \juj. (12.26) 
Now pre-multiply both sides by X to give 
1 
w XX (Xui) = \;(Xu;). (12.27) 
If we now define v; = Xu;, we obtain 
1 
gXX Vi = ÀiVi (12.28) 


which is an eigenvector equation for the N x N matrix N~'XX7. We see that this 
has the same N — 1 eigenvalues as the original covariance matrix (which itself has an 
additional D — N + 1 eigenvalues of value zero). Thus we can solve the eigenvector 
problem in spaces of lower dimensionality with computational cost O(N°) instead 
of O(D?). In order to determine the eigenvectors, we multiply both sides of (12.28) 
by XT to give 


(5x"x) (XTv,) = A(X" vi) (12.29) 


from which we see that (XT v;) is an eigenvector of S with eigenvalue ;. Note, 
however, that these eigenvectors need not be normalized. To determine the appropri- 
ate normalization, we re-scale u; « Xv, by a constant such that ||u,|| = 1, which, 
assuming v; has been normalized to unit length, gives 
1 T 

In summary, to apply this approach we first evaluate XXT and then find its eigen- 
vectors and eigenvalues and then compute the eigenvectors in the original data space 
using (12.30). 


Probabilistic PCA 


The formulation of PCA discussed in the previous section was based on a linear 
projection of the data onto a subspace of lower dimensionality than the original data 
space. We now show that PCA can also be expressed as the maximum likelihood 
solution of a probabilistic latent variable model. This reformulation of PCA, known 
as probabilistic PCA, brings several advantages compared with conventional PCA: 


e Probabilistic PCA represents a constrained form of the Gaussian distribution 
in which the number of free parameters can be restricted while still allowing 
the model to capture the dominant correlations in a data set. 


Section 12.2.2 


Section 12.2.3 


Section 8.1.4 


Section 8.2.2 


12.2. Probabilistic PCA 571 


We can derive an EM algorithm for PCA that is computationally efficient in 
situations where only a few leading eigenvectors are required and that avoids 
having to evaluate the data covariance matrix as an intermediate step. 


The combination of a probabilistic model and EM allows us to deal with miss- 
ing values in the data set. 


Mixtures of probabilistic PCA models can be formulated in a principled way 
and trained using the EM algorithm. 


Probabilistic PCA forms the basis for a Bayesian treatment of PCA in which 
the dimensionality of the principal subspace can be found automatically from 
the data. 


The existence of a likelihood function allows direct comparison with other 
probabilistic density models. By contrast, conventional PCA will assign a low 
reconstruction cost to data points that are close to the principal subspace even 
if they lie arbitrarily far from the training data. 


Probabilistic PCA can be used to model class-conditional densities and hence 
be applied to classification problems. 


The probabilistic PCA model can be run generatively to provide samples from 
the distribution. 


This formulation of PCA as a probabilistic model was proposed independently by 
Tipping and Bishop (1997, 1999b) and by Roweis (1998). As we shall see later, it is 
closely related to factor analysis (Basilevsky, 1994). 

Probabilistic PCA is a simple example of the linear-Gaussian framework, in 
which all of the marginal and conditional distributions are Gaussian. We can formu- 
late probabilistic PCA by first introducing an explicit latent variable z corresponding 
to the principal-component subspace. Next we define a Gaussian prior distribution 
p(z) over the latent variable, together with a Gaussian conditional distribution p(x|z) 
for the observed variable x conditioned on the value of the latent variable. Specifi- 
cally, the prior distribution over z is given by a zero-mean unit-covariance Gaussian 


p(z) = N (z|0, I). (12.31) 


Similarly, the conditional distribution of the observed variable x, conditioned on the 
value of the latent variable z, is again Gaussian, of the form 


p(x|z) = N(x|Wz + n, o°T) (12.32) 


in which the mean of x is a general linear function of z governed by the D x M 
matrix W and the D-dimensional vector jz. Note that this factorizes with respect to 
the elements of x, in other words this is an example of the naive Bayes model. As 
we shall see shortly, the columns of W span a linear subspace within the data space 
that corresponds to the principal subspace. The other parameter in this model is the 
scalar o? governing the variance of the conditional distribution. Note that there is no 
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Figure 12.9 An illustration of the generative view of the probabilistic PCA model for a two-dimensional data 
space and a one-dimensional latent space. An observed data point x is generated by first drawing a value 2 
for the latent variable from its prior distribution p{=) and then drawing a value for x from an isotropic Gaussian 
distribution (illustrated by the red circles) having mean w= + u and covariance oI. The green ellipses show the 
density contours for the marginal distribution p(x). 
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Exercise 12.7 


loss of generality in assuming a zero mean, unit covariance Gaussian for the latent 
distribution p(z) because a more general Gaussian distribution would give rise to an 
equivalent probabilistic model. 
We can view the probabilistic PCA model from a generative viewpoint in which 
a sampled value of the observed variable is obtained by first choosing a value for 
the latent variable and then sampling the observed variable conditioned on this la- 
tent value. Specifically, the D-dimensional observed variable x is defined by a lin- 
ear transformation of the M-dimensional latent variable z plus additive Gaussian 
‘noise’, so that 
x= Wza+pt+e (12.33) 


where z is an M-dimensional Gaussian latent variable, and € is a D-dimensional 
zero-mean Gaussian-distributed noise variable with covariance o7I. This generative 
process is illustrated in Figure 12.9, Note that this framework is based on a mapping 
from latent space to data space, in contrast to the more conventional view of PCA 
discussed above. The reverse mapping, from data space to the latent space, will be 
obtained shortly using Bayes’ theorem. 

Suppose we wish to determine the values of the parameters W, ys and o° using 
maximum likelihood. To write down the likelihood function, we need an expression 
for the marginal distribution p(x) of the observed variable. This is expressed, from 
the sum and product rules of probability, in the form 


p(x) = | ¢x\a)p(a) a. (12.34) 


Because this corresponds to a linear-Gaussian model, this marginal distribution is 
again Gaussian, and is given by 


p(x) = N(x|u, C) (1 


N 
to 
a 
= 
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where the D x D covariance matrix C is defined by 
C= WW! + ol. (12.36) 
This result can also be derived more directly by noting that the predictive distribution 


will be Gaussian and then evaluating its mean and covariance using (12.33). This 
gives 


ix} = E[Wz+u+e] = pu (12.37) 
covix] = E[|(Wz + e)(Wz + €)"] 
= E[Wzz"WT] + Eļee"] = WW" + o’°I (12.38) 


where we have used the fact that z and € are independent random variables and hence 
are uncorrelated. 

Intuitively, we can think of the distribution p(x) as being defined by taking an 
isotropic Gaussian ‘spray can’ and moving it across the principal subspace spraying 
Gaussian ink with density determined by o? and weighted by the prior distribution. 
The accumulated ink density gives rise to a ‘pancake’ shaped distribution represent- 
ing the marginal density p(x). 

The predictive distribution p(x) is governed by the parameters u, W, and o°. 
However, there is redundancy in this parameterization corresponding to rotations of 
the latent space coordinates. To see this, consider a matrix W = WR where R is 
an orthogonal matrix. Using the orthogonality property RRT = I, we see that the 


quantity WW" that appears in the covariance matrix C takes the form 
WW" = WRR'W! = WW” (12.39) 


and hence is independent of R. Thus there is a whole family of matrices W all of 
which give rise to the same predictive distribution. This invariance can be understood 
in terms of rotations within the latent space. We shall return to a discussion of the 
number of independent parameters in this model later. 

When we evaluate the predictive distribution, we require C~', which involves 
the inversion of a D x D matrix. The computation required to do this can be reduced 
by making use of the matrix inversion identity (C.7) to give 


Cl=o01l-a *WM IWT (12.40) 
where the M x M matrix M is defined by 
M=W'W 40°I. (12.41) 


Because we invert M rather than inverting C directly, the cost of evaluating C7! is 
reduced from O(D?) to O(M°). 

As well as the predictive distribution p(x), we will also require the posterior 
distribution p(z|x), which can again be written down directly using the result (2.116) 
for linear-Gaussian models to give 


p(2|x) = N (2|M~'W"(x — p),0° 7M). (12.42) 


Note that the posterior mean depends on x, whereas the posterior covariance is in- 
dependent of x. 
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Figure 12.10 The probabilistic PCA model for a data set of N obser- 


which each observation x» is associated with a value 7 


Zn Of the latent variable. 
aut i 


vations of x can be expressed as a directed graph in 5 € 


12.2.1 Maximum likelihood PCA 


We next consider the determination of the model parameters using maximum 
likelihood. Given a data set X = {x,,} of observed data points, the probabilistic 
PCA model can be expressed as a directed graph, as shown in Figure 12.10. The 
corresponding log likelihood function is given, from (12.35), by 


N 
Inp(X|u, W, 0°) = X` In p(xn|W, p, 0°) 
n=1 
ND ieee 
— pasi Amp i _ =i anna 
= -= ln(2r) A m|C| - 5 (n= (xn — ps). (12.43) 


Setting the derivative of the log likelihood with respect to jz equal to zero gives the 
expected result x = X where X is the data mean defined by (12.1). Back-substituting 
we can then write the log likelihood function in the form 


In p(X|W, ps, 0?) = -2 { D In(2m) + In |C| + Tr (C7'S) } (12.44) 


where S is the data covariance matrix defined by (12.3). Because the log likelihood 
is a quadratic function of yz, this solution represents the unique maximum, as can be 
confirmed by computing second derivatives. 

Maximization with respect to W and ø? is more complex but nonetheless has 
an exact closed-form solution. It was shown by Tipping and Bishop (1999b) that all 
of the stationary points of the log likelihood function can be written as 


Wmr = Uu (Lm — IPR (12.45) 


where U m isa D x M matrix whose columns are given by any subset (of size M) 
of the eigenvectors of the data covariance matrix S, the M x M diagonal matrix 
Ly has elements given by the corresponding eigenvalues ;, and R is an arbitrary 
M x M orthogonal matrix. 

Furthermore, Tipping and Bishop (1999b) showed that the maximum of the like- 
lihood function is obtained when the M eigenvectors are chosen to be those whose 
eigenvalues are the M largest (all other solutions being saddle points). A similar re- 
sult was conjectured independently by Roweis (1998), although no proof was given. 
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Again, we shall assume that the eigenvectors have been arranged in order of decreas- 
ing values of the corresponding eigenvalues, so that the M principal eigenvectors are 
uj,..., Ugg. In this case, the columns of W define the principal subspace of stan- 
dard PCA. The corresponding maximum likelihood solution for g? is then given by 


D 

1 

on = ere So di (12.46) 
i=M+1 

so that o?,,, is the average variance associated with the discarded dimensions. 

Because R is orthogonal, it can be interpreted as a rotation matrix in the M x M 
latent space. If we substitute the solution for W into the expression for C, and make 
use of the orthogonality property RR™ = I, we see that C is independent of R. 
This simply says that the predictive density is unchanged by rotations in the latent 
space as discussed earlier. For the particular case of R = I, we see that the columns 
of W are the principal component eigenvectors scaled by the variance parameters 
ài — o°. The interpretation of these scaling factors is clear once we recognize that 
for a convolution of independent Gaussian distributions (in this case the latent space 
distribution and the noise model) the variances are additive. Thus the variance A; 
in the direction of an eigenvector u; is composed of the sum of a contribution A; — 
a? from the projection of the unit-variance latent space distribution into data space 
through the corresponding column of W, plus an isotropic contribution of variance 
a? which is added in all directions by the noise model. 

It is worth taking a moment to study the form of the covariance matrix given 
by (12.36). Consider the variance of the predictive distribution along some direction 
specified by the unit vector v, where v'v = 1, which is given by v'Cv. First 
suppose that v is orthogonal to the principal subspace, in other words it is given by 
some linear combination of the discarded eigenvectors. Then v'U = 0 and hence 
v'Cv = o?. Thus the model predicts a noise variance orthogonal to the principal 
subspace, which, from (12.46), is just the average of the discarded eigenvalues. Now 
suppose that v = u; where u; is one of the retained eigenvectors defining the prin- 
cipal subspace. Then v’Cv = (A; — o?) + 0? = Xj. In other words, this model 
correctly captures the variance of the data along the principal axes, and approximates 
the variance in all remaining directions with a single average value o°. 

One way to construct the maximum likelihood density model would simply be 
to find the eigenvectors and eigenvalues of the data covariance matrix and then to 
evaluate W and g? using the results given above. In this case, we would choose 
R = I for convenience. However, if the maximum likelihood solution is found by 
numerical optimization of the likelihood function, for instance using an algorithm 
such as conjugate gradients (Fletcher, 1987; Nocedal and Wright, 1999; Bishop and 
Nabney, 2008) or through the EM algorithm, then the resulting value of R is es- 
sentially arbitrary. This implies that the columns of W need not be orthogonal. If 
an orthogonal basis is required, the matrix W can be post-processed appropriately 
(Golub and Van Loan, 1996). Alternatively, the EM algorithm can be modified in 
such a way as to yield orthonormal principal directions, sorted in descending order 
of the corresponding eigenvalues, directly (Ahn and Oh, 2003). 
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The rotational invariance in latent space represents a form of statistical noniden- 
tifiability, analogous to that encountered for mixture models in the case of discrete 
latent variables. Here there is a continuum of parameters all of which lead to the 
same predictive density, in contrast to the discrete nonidentifiability associated with 
component re-labelling in the mixture setting. 

If we consider the case of M = D, so that there is no reduction of dimension- 
ality, then Uy = U and Ly = L. Making use of the orthogonality properties 
UUT = I and RR’ = I, we see that the covariance C of the marginal distribution 
for x becomes 


C = U(L— o'I} PRRI (L — 0°I)/?U" +0°I = ULUT =S (12.47) 


and so we obtain the standard maximum likelihood solution for an unconstrained 
Gaussian distribution in which the covariance matrix is given by the sample covari- 
ance. 

Conventional PCA is generally formulated as a projection of points from the D- 
dimensional data space onto an M-dimensional linear subspace. Probabilistic PCA, 
however, is most naturally expressed as a mapping from the latent space into the data 
space via (12.33). For applications such as visualization and data compression, we 
can reverse this mapping using Bayes’ theorem. Any point x in data space can then 
be summarized by its posterior mean and covariance in latent space. From (12.42) 
the mean is given by 

E[z|x] = M`'WiL(x — X) (12.48) 


where M is given by (12.41). This projects to a point in data space given by 
WE[z|x] + m. (12.49) 


Note that this takes the same form as the equations for regularized linear regression 
and is a consequence of maximizing the likelihood function for a linear Gaussian 
model. Similarly, the posterior covariance is given from (12.42) by o?°M™! and is 
independent of x. 

If we take the limit o? — 0, then the posterior mean reduces to 


(WiLWm) Wai (x — X) (12.50) 


which represents an orthogonal projection of the data point onto the latent space, 
and so we recover the standard PCA model. The posterior covariance in this limit is 
zero, however, and the density becomes singular. For o? > 0, the latent projection 
is shifted towards the origin, relative to the orthogonal projection. 

Finally, we note that an important role for the probabilistic PCA model is in 
defining a multivariate Gaussian distribution in which the number of degrees of free- 
dom, in other words the number of independent parameters, can be controlled whilst 
still allowing the model to capture the dominant correlations in the data. Recall 
that a general Gaussian distribution has D(D + 1)/2 independent parameters in its 
covariance matrix (plus another D parameters in its mean). Thus the number of 
parameters scales quadratically with D and can become excessive in spaces of high 
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dimensionality. If we restrict the covariance matrix to be diagonal, then it has only D 
independent parameters, and so the number of parameters now grows linearly with 
dimensionality. However, it now treats the variables as if they were independent and 
hence can no longer express any correlations between them. Probabilistic PCA pro- 
vides an elegant compromise in which the M most significant correlations can be 
captured while still ensuring that the total number of parameters grows only linearly 
with D. We can see this by evaluating the number of degrees of freedom in the 
PPCA model as follows. The covariance matrix C depends on the parameters W, 
which has size D x M, and o?, giving a total parameter count of DM +1. However, 
we have seen that there is some redundancy in this parameterization associated with 
rotations of the coordinate system in the latent space. The orthogonal matrix R. that 
expresses these rotations has size M x M. In the first column of this matrix there are 
M — 1 independent parameters, because the column vector must be normalized to 
unit length. In the second column there are M — 2 independent parameters, because 
the column must be normalized and also must be orthogonal to the previous column, 
and so on. Summing this arithmetic series, we see that R. has a total of M(M —1)/2 
independent parameters. Thus the number of degrees of freedom in the covariance 
matrix C is given by 

DM +1—M(M —-1)/2. (12.51) 


The number of independent parameters in this model therefore only grows linearly 
with D, for fixed M. If we take M = D — 1, then we recover the standard result 
for a full covariance Gaussian. In this case, the variance along D — 1 linearly in- 
dependent directions is controlled by the columns of W, and the variance along the 
remaining direction is given by a7. If M = 0, the model is equivalent to the isotropic 
covariance case. 


12.2.2 EM algorithm for PCA 


As we have seen, the probabilistic PCA model can be expressed in terms of a 
marginalization over a continuous latent space z in which for each data point Xn, 
there is a corresponding latent variable z,,. We can therefore make use of the EM 
algorithm to find maximum likelihood estimates of the model parameters. This may 
seem rather pointless because we have already obtained an exact closed-form so- 
lution for the maximum likelihood parameter values. However, in spaces of high 
dimensionality, there may be computational advantages in using an iterative EM 
procedure rather than working directly with the sample covariance matrix. This EM 
procedure can also be extended to the factor analysis model, for which there is no 
closed-form solution. Finally, it allows missing data to be handled in a principled 
way. 

We can derive the EM algorithm for probabilistic PCA by following the general 
framework for EM. Thus we write down the complete-data log likelihood and take 
its expectation with respect to the posterior distribution of the latent distribution 
evaluated using ‘old’ parameter values. Maximization of this expected complete- 
data log likelihood then yields the ‘new’ parameter values. Because the data points 
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are assumed independent, the complete-data log likelihood function takes the form 


N 
Inp (X, Z|, W, 07) = 5 {In p(Xn|Zn) + ln p(Zn)} (12.52) 


n=l 


where the nt? row of the matrix Z is given by Zn. We already know that the exact 
maximum likelihood solution for yz is given by the sample mean X defined by (12.1), 
and it is convenient to substitute for ys at this stage. Making use of the expressions 
(12.31) and (12.32) for the latent and conditional distributions, respectively, and tak- 
ing the expectation with respect to the posterior distribution over the latent variables, 
we obtain 


N 


D 1 
Elin p (X, Z|, W, o”)] = — B [2 ln(2r0°) + ai (E[znzn]) 
1 2 l E TWT 
+552 [xn — Hl z2 [Zn] (Xn — H) 
1 
Foai (Elenazjw*w) } i (12.53) 


Note that this depends on the posterior distribution only through the sufficient statis- 
tics of the Gaussian. Thus in the E step, we use the old parameter values to evaluate 


E[zn] = M`’WT(x, — X) (12.54) 
E[2nzi] = o?M7!+Elz,|E[z,]7 (12.55) 


n 
which follow directly from the posterior distribution (12.42) together with the stan- 
dard result E[znz}] = cov[zn] + E[z,]E[z,]". Here M is defined by (12.41). 
In the M step, we maximize with respect to W and o°, keeping the posterior 
statistics fixed. Maximization with respect to o? is straightforward. For the maxi- 
mization with respect to W we make use of (C.24), and obtain the M-step equations 


N N a 
Wrew = Sots, — se" pe) (12.56) 


n=1 
1 N 
2 = 7l TyyT _> 
Trew = ND 2a {le F|? = 2B [en] Wrrew (Xn — 3) 
n= 
+Tr (Elznz |W w Wnew) }. (12.57) 


The EM algorithm for probabilistic PCA proceeds by initializing the parameters 
and then alternately computing the sufficient statistics of the latent space posterior 
distribution using (12.54) and (12.55) in the E step and revising the parameter values 
using (12.56) and (12.57) in the M step. 

One of the benefits of the EM algorithm for PCA is computational efficiency 
for large-scale applications (Roweis, 1998). Unlike conventional PCA based on an 
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eigenvector decomposition of the sample covariance matrix, the EM approach is 
iterative and so might appear to be less attractive. However, each cycle of the EM 
algorithm can be computationally much more efficient than conventional PCA in 
spaces of high dimensionality. To see this, we note that the eigendecomposition of 
the covariance matrix requires O(D*) computation. Often we are interested only 
in the first M eigenvectors and their corresponding eigenvalues, in which case we 
can use algorithms that are O(M D7). However, the evaluation of the covariance 
matrix itself takes O(N D?) computations, where N is the number of data points. 
Algorithms such as the snapshot method (Sirovich, 1987), which assume that the 
eigenvectors are linear combinations of the data vectors, avoid direct evaluation of 
the covariance matrix but are O( N°3) and hence unsuited to large data sets. The EM 
algorithm described here also does not construct the covariance matrix explicitly. 
Instead, the most computationally demanding steps are those involving sums over 
the data set that are O(N DM). For large D, and M < D, this can be a significant 
saving compared to O(N D?) and can offset the iterative nature of the EM algorithm. 

Note that this EM algorithm can be implemented in an on-line form in which 
each D-dimensional data point is read in and processed and then discarded before 
the next data point is considered. To see this, note that the quantities evaluated in 
the E step (an M-dimensional vector and an M x M matrix) can be computed for 
each data point separately, and in the M step we need to accumulate sums over data 
points, which we can do incrementally. This approach can be advantageous if both 
N and D are large. 

Because we now have a fully probabilistic model for PCA, we can deal with 
missing data, provided that it is missing at random, by marginalizing over the dis- 
tribution of the unobserved variables. Again these missing values can be treated 
using the EM algorithm. We give an example of the use of this approach for data 
visualization in Figure 12.11. 

Another elegant feature of the EM approach is that we can take the limit o? — 0, 
corresponding to standard PCA, and still obtain a valid EM-like algorithm (Roweis, 
1998). From (12.55), we see that the only quantity we need to compute in the E step 
is E[z,,]. Furthermore, the M step is simplified because M = WTW. To emphasize 


the simplicity of the algorithm, let us define X to be a matrix of size N x D whose 
n*® row is given by the vector x,, — X and similarly define Q to be a matrix of size 
D x M whose n™ row is given by the vector E[z,,]. The E step (12.54) of the EM 


algorithm for PCA then becomes 


2 = (W5yWoa) )WiaX (12.58) 
and the M step (12.56) takes the form 
Wrew = XTAT(AANT)“!. (12.59) 


Again these can be implemented in an on-line form. These equations have a simple 
interpretation as follows. From our earlier discussion, we see that the E step involves 
an orthogonal projection of the data points onto the current estimate for the principal 
subspace. Correspondingly, the M step represents a re-estimation of the principal 
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Figure 12.11 Probabilistic PCA visualization of a portion of the oil flow data set for the first 100 data points. The 
left-hand plot shows the posterior mean projections of the data points on the principal subspace. The right-hand 
plot is obtained by first randomly omitting 30% of the variable values and then using EM to handle the missing 
values. Note that each data point then has at least one missing measurement but that the plot is very similar to 
the one obtained without missing values. 


Exercise 12.17 


subspace to minimize the squared reconstruction error in which the projections are 
fixed. 

We can give a simple physical analogy for this EM algorithm, which is easily 
visualized for D = 2 and M = 1. Consider a collection of data points in two 
dimensions, and let the one-dimensional principal subspace be represented by a solid 
rod. Now attach each data point to the rod via a spring obeying Hooke’s law (stored 
energy is proportional to the square of the spring’s length). In the E step, we keep 
the rod fixed and allow the attachment points to slide up and down the rod so as to 
minimize the energy. This causes each attachment point (independently) to position 
itself at the orthogonal projection of the corresponding data point onto the rod. In 
the M step, we keep the attachment points fixed and then release the rod and allow it 
to move to the minimum energy position. The E and M steps are then repeated until 
a suitable convergence criterion is satisfied, as is illustrated in Figure 12.12. 


12.2.3 Bayesian PCA 


So far in our discussion of PCA, we have assumed that the value A for the 
dimensionality of the principal subspace is given. In practice, we must choose a 
suitable value according to the application. For visualization, we generally choose 
M = 2, whereas for other applications the appropriate choice for M may be less 
clear. One approach is to plot the eigenvalue spectrum for the data set, analogous 
to the example in Figure 12.4 for the off-line digits data set, and look to see if the 
eigenvalues naturally form two groups comprising a set of small values separated by 
a significant gap from a set of relatively large values, indicating a natural choice for 
M. In practice, such a gap is often not seen. 
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Figure 12.12 Synthetic data illustrating the EM algorithm for PCA defined by (12.58) and (12.59). (a) A data 
set X with the data points shown in green, together with the true principal components (shown as eigenvectors 
scaled by the square roots of the eigenvalues). (b) Initial configuration of the principal subspace defined by W, 
shown in red, together with the projections of the latent points Z into the data space, given by ZW", shown in 
cyan. (c) After one M step, the latent space has been updated with Z held fixed. (d) After the successive E step, 
the values of Z have been updated, giving orthogonal projections, with W held fixed. (e) After the second M 
step. (f) After the second E step. 


Section 1.3 


Because the probabilistic PCA model has a well-defined likelihood function, we 
could employ cross-validation to determine the value of dimensionality by selecting 
the largest log likelihood on a validation data set. Such an approach, however, can 
become computationally costly, particularly if we consider a probabilistic mixture 
of PCA models (Tipping and Bishop, 1999a) in which we seek to determine the 
appropriate dimensionality separately for each component in the mixture. 

Given that we have a probabilistic formulation of PCA, it seems natural to seek 
a Bayesian approach to model selection. To do this, we need to marginalize out 
the model parameters jz, W, and g? with respect to appropriate prior distributions. 
This can be done by using a variational framework to approximate the analytically 
intractable marginalizations (Bishop, 1999b). The marginal likelihood values, given 
by the variational lower bound, can then be compared for a range of different values 
of M and the value giving the largest marginal likelihood selected. 

Here we consider a simpler approach introduced by based on the evidence ap- 
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Figure 12.13 Probabilistic graphical model for Bayesian PCA in 
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which the distribution over the parameter matrix W 


h 2 
is governed by a vector a of hyperparameters. g 


proximation, which is appropriate when the number of data points is relatively large 
and the corresponding posterior distribution is tightly peaked (Bishop, 1999a). It 
involves a specific choice of prior over W that allows surplus dimensions in the 
principal subspace to be pruned out of the model. This corresponds to an example of 
automatic relevance determination, or ARD, discussed in Section 7.2.2. Specifically, 
we define an independent Gaussian prior over each column of W, which represent 
the vectors defining the principal subspace. Each such Gaussian has an independent 
variance governed by a precision hyperparameter œ; so that 


M 

wash e a a 12.60 

p a)=[] On exp z%Wi Wi (12.60) 
i=l 


where w; is the i” column of W. The resulting model can be represented using the 
directed graph shown in Figure 12.13. 

The values for a; will be found iteratively by maximizing the marginal likeli- 
hood function in which W has been integrated out. As a result of this optimization, 
some of the a; may be driven to infinity, with the corresponding parameters vec- 
tor w; being driven to zero (the posterior distribution becomes a delta function at 
the origin) giving a sparse solution. The effective dimensionality of the principal 
subspace is then determined by the number of finite œ; values, and the correspond- 
ing vectors w; can be thought of as ‘relevant’ for modelling the data distribution. 
In this way, the Bayesian approach is automatically making the trade-off between 
improving the fit to the data, by using a larger number of vectors w; with their cor- 
responding eigenvalues À; each tuned to the data, and reducing the complexity of 
the model by suppressing some of the w; vectors. The origins of this sparsity were 
discussed earlier in the context of relevance vector machines. 

The values of a; are re-estimated during training by maximizing the log marginal 
likelihood given by 


p(Xla, u, o’) = J X1W.1,02)0(Wle) dW (12.61) 


where the log of p(X|W , p, 0”) is given by (12.43). Note that for simplicity we also 
treat ys and g? as parameters to be estimated, rather than defining priors over these 
parameters. 


Section 4.4 


Section 3.5.3 
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Because this integration is intractable, we make use of the Laplace approxima- 
tion. If we assume that the posterior distribution is sharply peaked, as will occur for 
sufficiently large data sets, then the re-estimation equations obtained by maximizing 
the marginal likelihood with respect to a; take the simple form 


pew = (12.62) 


which follows from (3.98), noting that the dimensionality of w; is D. These re- 
estimations are interleaved with the EM algorithm updates for determining W and 
a”. The E-step equations are again given by (12.54) and (12.55). Similarly, the M- 
step equation for g? is again given by (12.57). The only change is to the M-step 
equation for W, which is modified to give 


N N -1 
Wrew = [S - sel 2 Elznz:] + a (12.63) 


n=1 n=1 


where A = diag(a;). The value of p is given by the sample mean, as before. 

If we choose M = D — 1 then, if all a; values are finite, the model represents 
a full-covariance Gaussian, while if all the œ; go to infinity the model is equivalent 
to an isotropic Gaussian, and so the model can encompass all permissible values for 
the effective dimensionality of the principal subspace. It is also possible to consider 
smaller values of M, which will save on computational cost but which will limit 
the maximum dimensionality of the subspace. A comparison of the results of this 
algorithm with standard probabilistic PCA is shown in Figure 12.14. 

Bayesian PCA provides an opportunity to illustrate the Gibbs sampling algo- 
rithm discussed in Section 11.3. Figure 12.15 shows an example of the samples 
from the hyperparameters In a; for a data set in D = 4 dimensions in which the di- 
mensionality of the latent space is M = 3 but in which the data set is generated from 
a probabilistic PCA model having one direction of high variance, with the remaining 
directions comprising low variance noise. This result shows clearly the presence of 
three distinct modes in the posterior distribution. At each step of the iteration, one of 
the hyperparameters has a small value and the remaining two have large values, so 
that two of the three latent variables are suppressed. During the course of the Gibbs 
sampling, the solution makes sharp transitions between the three modes. 

The model described here involves a prior only over the matrix W. A fully 
Bayesian treatment of PCA, including priors over p, c°, and a, and solved us- 
ing variational methods, is described in Bishop (1999b). For a discussion of vari- 
ous Bayesian approaches to determining the appropriate dimensionality for a PCA 
model, see Minka (2001c). 


12.2.4 Factor analysis 


Factor analysis is a linear-Gaussian latent variable model that is closely related 
to probabilistic PCA. Its definition differs from that of probabilistic PCA only in that 
the conditional distribution of the observed variable x given the latent variable z is 
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Figure 12.14 


‘Hinton’ diagrams of the matrix W in which each element of the matrix is depicted as 
a square (white for positive and black for negative values) whose area is proportional 
to the magnitude of that element. The synthetic data set comprises 300 data points in 
D = 10 dimensions sampled from a Gaussian distribution having standard deviation 1.0 
in 3 directions and standard deviation 0.5 in the remaining 7 directions for a data set in 
D = 10 dimensions having M = 3 directions with larger variance than the remaining 7 
directions. The left-hand plot shows the result from maximum likelihood probabilistic PCA, 
and the left-hand plot shows the corresponding result from Bayesian PCA. We see how 
the Bayesian model is able to discover the appropriate dimensionality by suppressing the 
6 surplus degrees of freedom. 


taken to have a diagonal rather than an isotropic covariance so that 
p(x|z) = N(x|Wz + p, ©) (12.64) 


where Ẹ is a D x D diagonal matrix. Note that the factor analysis model, in common 
with probabilistic PCA, assumes that the observed variables x,,..., £p are indepen- 
dent, given the latent variable z. In essence, the factor analysis model is explaining 
the observed covariance structure of the data by representing the independent vari- 
ance associated with each coordinate in the matrix W and capturing the covariance 
between variables in the matrix W. In the factor analysis literature, the columns 
of W, which capture the correlations between observed variables, are called factor 
loadings, and the diagonal elements of W, which represent the independent noise 
variances for each of the variables, are called uniquenesses. 

The origins of factor analysis are as old as those of PCA, and discussions of 
factor analysis can be found in the books by Everitt (1984), Bartholomew (1987), 
and Basilevsky (1994). Links between factor analysis and PCA were investigated 
by Lawley (1953) and Anderson (1963) who showed that at stationary points of 
the likelihood function, for a factor analysis model with W = o7I, the columns of 
W are scaled eigenvectors of the sample covariance matrix, and g? is the average 
of the discarded eigenvalues. Later, Tipping and Bishop (1999b) showed that the 
maximum of the log likelihood function occurs when the eigenvectors comprising 
W are chosen to be the principal eigenvectors. 

Making use of (2.115), we see that the marginal distribution for the observed 


Figure 12.15 Gibbs sampling for Bayesian 
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PCA showing plots of Ina; 
versus iteration number for 
three a values, showing 
transitions between the 
three modes of the posterior 
distribution. 


variable is given by p(x) = N (x|, C) where now 
C=WW'+W. (12.65) 


As with probabilistic PCA, this model is invariant to rotations in the latent space. 

Historically, factor analysis has been the subject of controversy when attempts 
have been made to place an interpretation on the individual factors (the coordinates 
in z-space), which has proven problematic due to the nonidentifiability of factor 
analysis associated with rotations in this space. From our perspective, however, we 
shall view factor analysis as a form of latent variable density model, in which the 
form of the latent space is of interest but not the particular choice of coordinates 
used to describe it. If we wish to remove the degeneracy associated with latent space 
rotations, we must consider non-Gaussian latent variable distributions, giving rise to 
independent component analysis (ICA) models. 

We can determine the parameters u, W, and in the factor analysis model by 
maximum likelihood. The solution for p is again given by the sample mean. How- 
ever, unlike probabilistic PCA, there is no longer a closed-form maximum likelihood 
solution for W, which must therefore be found iteratively. Because factor analysis is 
a latent variable model, this can be done using an EM algorithm (Rubin and Thayer, 
1982) that is analogous to the one used for probabilistic PCA. Specifically, the E-step 
equations are given by 


Elz,| = GW'©'(x, — X) (12.66) 
E[z,z,;] = G+Efz,JE(z,]' (12.67) 

where we have defined 7 
G = (I+ WTW)! (12.68) 


Note that this is expressed in a form that involves inversion of matrices of size M x M 
rather than D x D (except for the D x D diagonal matrix W whose inverse is trivial 
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12.3. 


to compute in O(D) steps), which is convenient because often M < D. Similarly, 
the M-step equations take the form 


N N =I 
wee = ead Yee (12.69) 


N 
gew = diag fs “= Wrew 3 7 [Zn] (Xn gi =| (12.70) 


where the ‘diag’ operator sets all of the nondiagonal elements of a matrix to zero. A 
Bayesian treatment of the factor analysis model can be obtained by a straightforward 
application of the techniques discussed in this book. 

Another difference between probabilistic PCA and factor analysis concerns their 
different behaviour under transformations of the data set. For PCA and probabilis- 
tic PCA, if we rotate the coordinate system in data space, then we obtain exactly 
the same fit to the data but with the W matrix transformed by the corresponding 
rotation matrix. However, for factor analysis, the analogous property is that if we 
make a component-wise re-scaling of the data vectors, then this is absorbed into a 
corresponding re-scaling of the elements of Y. 


Kernel PCA 


In Chapter 6, we saw how the technique of kernel substitution allows us to take an 
algorithm expressed in terms of scalar products of the form xTx’ and generalize 
that algorithm by replacing the scalar products with a nonlinear kernel. Here we 
apply this technique of kernel substitution to principal component analysis, thereby 
obtaining a nonlinear generalization called kernel PCA (Scholkopf et al., 1998). 
Consider a data set {Xn } of observations, where n = 1,..., N, in a space of 
dimensionality D. In order to keep the notation uncluttered, we shall assume that 
we have already subtracted the sample mean from each of the vectors x, so that 
Xn Xn = 0. The first step is to express conventional PCA in such a form that the 
data vectors {x,,} appear only in the form of the scalar products xf xm. Recall that 
the principal components are defined by the eigenvectors u; of the covariance matrix 


Su; = Aju, (12.71) 
where i = 1,..., D. Here the D x D sample covariance matrix S is defined by 
Pe x 
=F ` XnXT, (12.72) 
n=1 


and the eigenvectors are normalized such that uu; = 1. 
Now consider a nonlinear transformation (x) into an M-dimensional feature 
space, so that each data point x, is thereby projected onto a point @(x,,). We can 
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Figure 12.16 Schematic illustration of kernel PCA. A data set in the original data space (left-hand plot) is 
projected by a nonlinear transformation @(x) into a feature space (right-hand plot). By performing PCA in the 
feature space, we obtain the principal components, of which the first is shown in blue and is denoted by the 
vector vı. The green lines in feature space indicate the linear projections onto the first principal component, 
which correspond to nonlinear projections in the original data space. Note that in general it is not possible to 
represent the nonlinear principal component by a vector in x space. 


now perform standard PCA in the feature space, which implicitly defines a nonlinear 
principal component model in the original data space, as illustrated in Figure 12.16. 

For the moment, let us assume that the projected data set also has zero mean, 
so that 5°, @(x,,) = 0. We shall return to this point shortly. The M x M sample 
covariance matrix in feature space is given by 


N 
C= $ 2 olxn)ol(xn) (12.73) 
and its eigenvector expansion is defined by 
Cv; = \;Vi (12.74) 
i = 1,..., M. Our goal is to solve this eigenvalue problem without having to work 


explicitly in the feature space. From the definition of C, the eigenvector equations 
tells us that v; satisfies 


N 
DES {Ølxn) vi} = Aivi; (12.75) 


n=1 


and so we see that (provided A; > 0) the vector v; is given by a linear combination 
of the @(x,,) and so can be written in the form 


N 


vy, = 5 Ain @(Xn). (12.76) 


n=1 
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Substituting this expansion back into the eigenvector equation, we obtain 


pË N N 
T Do Xn) olan)" J | dimb(%m) = Ai) | din (Kn): (12.77) 
n=1 m=1 n=1 


The key step is now to express this in terms of the kernel function k(x,,, Xm) = 
(Xn)' (Xm), which we do by multiplying both sides by $(x;)" to give 


m 


N 
E DO k(x xn) J imk (Xn, Xm) = xÝ ank XI, Xn): (12.78) 
n=1 


mal n=l 
This can be written in matrix notation as 
K?a; = åN Ka; (12.79) 


where a; is an N-dimensional column vector with elements an; for n = 1,..., N. 
We can find solutions for a; by solving the following eigenvalue problem 


in which we have removed a factor of K from both sides of (12.79). Note that 
the solutions of (12.79) and (12.80) differ only by eigenvectors of K having zero 
eigenvalues that do not affect the principal components projection. 

The normalization condition for the coefficients a; is obtained by requiring that 
the eigenvectors in feature space be normalized. Using (12.76) and (12.80), we have 


N N 

1 = vv; = 5 5 Qin@im@(Xn)' (Xm) = a; Ka; =A;Naja;. (12.81) 
n=1 m=1 

Having solved the eigenvector problem, the resulting principal component pro- 


jections can then also be cast in terms of the kernel function so that, using (12.76), 
the projection of a point x onto eigenvector 7 is given by 


N 
y(x) = O(x)"vi = $ aind(x 7s dink(X,%n) (12.82) 
n=l n=1 


and so again is expressed in terms of the kernel function. 

In the original D-dimensional x space there are D orthogonal eigenvectors and 
hence we can find at most D linear principal components. The dimensionality M 
of the feature space, however, can be much larger than D (even infinite), and thus 
we can find a number of nonlinear principal components that can exceed D. Note, 
however, that the number of nonzero eigenvalues cannot exceed the number N of 
data points, because (even if Af > N) the covariance matrix in feature space has 
rank at most equal to N. This is reflected in the fact that kernel PCA involves the 
eigenvector expansion of the N x N matrix K. 
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So far we have assumed that the projected data set given by (xn) has zero 
mean, which in general will not be the case. We cannot simply compute and then 
subtract off the mean, since we wish to avoid working directly in feature space, and 
so again, we formulate the algorithm purely in terms of the kernel function. The 


projected data points after centralizing, denoted (Xn), are given by 


Plxn) = O(n) - = Eo (12.83) 


and the corresponding elements of the Gram matrix are given by 
Kam = #(Xn)' G(Xm) 
= (xn) (Xm) — >> b(Xn)" 


l=1 j=1 l=1 


1 N 1 N N 
ay D kCn xt) + a DD kx). (12.84) 


This can be expressed in matrix notation as 
K = K - 1nK— K1y +1yK1y (12.85) 


where 1y denotes the N x N matrix in which every element takes the value 1/N. 


Thus we can evaluate K using only the kernel function and then use K to determine 
the eigenvalues and eigenvectors. Note that the standard PCA algorithm is recovered 
as a special case if we use a linear kernel k(x, x’) = xx’. Figure 12.17 shows an 
example of kernel PCA applied to a synthetic data set (Scholkopf et al., 1998). Here 
a ‘Gaussian’ kernel of the form 


k(x, x’) = exp(—||x — x’||?/0.1) (12.86) 


is applied to a synthetic data set. The lines correspond to contours along which the 
projection onto the corresponding principal component, defined by 


N 


(x) vi = X aink(X, Xn) (12.87) 


n=1 


is constant. 
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Eigenvatue=21.72 


Eigenvalue=21.65 Eigenvalue=3.93 


Eigenvalue3.09 Eigenvalue=2.53 


Figure 12.17 Example of kernel PCA, with a Gaussian kernel applied to a synthetic data set in two dimensions, 
showing the first eight eigenfunctions along with their eigenvalues. The contours are lines along which the 
projection onto the corresponding principal component is constant. Note how the first two eigenvectors separate 
the three clusters, the next three eigenvectors split each of the cluster into halves, and the following three 
eigenvectors again split the clusters into halves along directions orthogonal to the previous splits. 


One obvious disadvantage of kernel PCA is that it involves finding the eigenvec- 
tors of the NV x N matrix K rather than the D x D matrix S of conventional linear 
PCA, and so in practice for large data sets approximations are often used. 

Finally, we note that in standard linear PCA, we often retain some reduced num- 
ber L < D of eigenvectors and then approximate a data vector x,, by its projection 
Xn onto the L-dimensional principal subspace, defined by 


L 


Ta > (x"u;) uy. (12.88) 


i=l 


In kernel PCA, this will in general not be possible. To see this, note that the map- 
ping @(x) maps the D-dimensional x space into a D-dimensional manifold in the 
M-dimensional feature space @. The vector x is known as the pre-image of the 
corresponding point @(x). However, the projection of points in feature space onto 
the linear PCA subspace in that space will typically not lie on the nonlinear D- 
dimensional manifold and so will not have a corresponding pre-image in data space. 
Techniques have therefore been proposed for finding approximate pre-images (Bakir 
et al., 2004). 
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Nonlinear Latent Variable Models 


In this chapter, we have focussed on the simplest class of models having continuous 
latent variables, namely those based on linear-Gaussian distributions. As well as 
having great practical importance, these models are relatively easy to analyse and 
to fit to data and can also be used as components in more complex models. Here 
we consider briefly some generalizations of this framework to models that are either 
nonlinear or non-Gaussian, or both. 

In fact, the issues of nonlinearity and non-Gaussianity are related because a 
general probability density can be obtained from a simple fixed reference density, 
such as a Gaussian, by making a nonlinear change of variables. This idea forms the 
basis of several practical latent variable models as we shall see shortly. 


12.4.1 Independent component analysis 


We begin by considering models in which the observed variables are related 
linearly to the latent variables, but for which the latent distribution is non-Gaussian. 
An important class of such models, known as independent component analysis, or 
ICA, arises when we consider a distribution over the latent variables that factorizes, 
so that 


M 
p(z) = | [ v(z)). (12.89) 
j=l 


To understand the role of such models, consider a situation in which two people 
are talking at the same time, and we record their voices using two microphones. 
If we ignore effects such as time delay and echoes, then the signals received by 
the microphones at any point in time will be given by linear combinations of the 
amplitudes of the two voices. The coefficients of this linear combination will be 
constant, and if we can infer their values from sample data, then we can invert the 
mixing process (assuming it is nonsingular) and thereby obtain two clean signals 
each of which contains the voice of just one person. This is an example of a problem 
called blind source separation in which ‘blind’ refers to the fact that we are given 
only the mixed data, and neither the original sources nor the mixing coefficients are 
observed (Cardoso, 1998). 

This type of problem is sometimes addressed using the following approach 
(MacKay, 2003) in which we ignore the temporal nature of the signals and treat the 
successive samples as i.i.d. We consider a generative model in which there are two 
latent variables corresponding to the unobserved speech signal amplitudes, and there 
are two observed variables given by the signal values at the microphones. The latent 
variables have a joint distribution that factorizes as above, and the observed variables 
are given by a linear combination of the latent variables. There is no need to include 
a noise distribution because the number of latent variables equals the number of ob- 
served variables, and therefore the marginal distribution of the observed variables 
will not in general be singular, so the observed variables are simply deterministic 
linear combinations of the latent variables. Given a data set of observations, the 


592 12. CONTINUOUS LATENT VARIABLES 


Exercise 12,29 


likelihood function for this model is a function of the coefficients in the linear com- 
bination. The log likelihood can be maximized using gradient-based optimization 
giving rise to a particular version of independent component analysis. 

The success of this approach requires that the latent variables have non-Gaussian 
distributions. To see this, recall that in probabilistic PCA (and in factor analysis) the 
latent-space distribution is given by a zero-mean isotropic Gaussian. The model 
therefore cannot distinguish between two different choices for the latent variables 
where these differ simply by a rotation in latent space. This can be verified directly 
by noting that the marginal density (12.35), and hence the likelihood function, is 
unchanged if we make the transformation W — WR where R is an orthogonal 
matrix satisfying RRT = I, because the matrix C given by (12.36) is itself invariant. 
Extending the model to allow more general Gaussian latent distributions does not 
change this conclusion because, as we have seen, such a model is equivalent to the 
zero-mean isotropic Gaussian latent variable model. 

Another way to see why a Gaussian latent variable distribution in a linear model 
is insufficient to find independent components is to note that the principal compo- 
nents represent a rotation of the coordinate system in data space such as to diagonal- 
ize the covariance matrix, so that the data distribution in the new coordinates is then 
uncorrelated. Although zero correlation is a necessary condition for independence 
it is not, however, sufficient. In practice, a common choice for the latent-variable 
distribution is given by 


1 1 
plz) = mcosh(z;)  n(e%i + e77) Ge) 


which has heavy tails compared to a Gaussian, reflecting the observation that many 
real-world distributions also exhibit this property. 

The original ICA model (Bell and Sejnowski, 1995) was based on the optimiza- 
tion of an objective function defined by information maximization. One advantage 
of a probabilistic latent variable formulation is that it helps to motivate and formu- 
late generalizations of basic ICA. For instance, independent factor analysis (Attias, 
1999a) considers a model in which the number of latent and observed variables can 
differ, the observed variables are noisy, and the individual latent variables have flex- 
ible distributions modelled by mixtures of Gaussians. The log likelihood for this 
model is maximized using EM, and the reconstruction of the latent variables is ap- 
proximated using a variational approach. Many other types of model have been 
considered, and there is now a huge literature on ICA and its applications (Jutten 
and Herault, 1991; Comon et al., 1991; Amari et al., 1996; Pearlmutter and Parra, 
1997; Hyvarinen and Oja, 1997; Hinton et al., 2001; Miskin and MacKay, 2001; 
Hojen-Sorensen et al., 2002; Choudrey and Roberts, 2003; Chan et al., 2003; Stone, 
2004). 


12.4.2 Autoassociative neural networks 


In Chapter 5 we considered neural networks in the context of supervised learn- 
ing, where the role of the network is to predict the output variables given values 
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Figure 12.18 An autoassociative multilayer perceptron having 
two layers of weights. Such a network is trained to 
map input vectors onto themselves by minimiza- 
tion of a sum-of-squares error. Even with non- 
linear units in the hidden layer, such a network 
is equivalent to linear principal component anal- 
ysis. Links representing bias parameters have 
been omitted for clarity. 


for the input variables. However, neural networks have also been applied to un- 
supervised learning where they have been used for dimensionality reduction. This 
is achieved by using a network having the same number of outputs as inputs, and 
optimizing the weights so as to minimize some measure of the reconstruction error 
between inputs and outputs with respect to a set of training data. 

Consider first a multilayer perceptron of the form shown in Figure 12.18, hav- 
ing D inputs, D output units and M hidden units, with M < D. The targets used 
to train the network are simply the input vectors themselves, so that the network 
is attempting to map each input vector onto itself. Such a network is said to form 
an autoassociative mapping. Since the number of hidden units is smaller than the 
number of inputs, a perfect reconstruction of all input vectors is not in general pos- 
sible. We therefore determine the network parameters w by minimizing an error 
function which captures the degree of mismatch between the input vectors and their 
reconstructions. In particular, we shall choose a sum-of-squares error of the form 


N 
1 
E(w) = DD lly (Xn) w) — xnl. (12.91) 
n=l 


If the hidden units have linear activations functions, then it can be shown that the 
error function has a unique global minimum, and that at this minimum the network 
performs a projection onto the M/-dimensional subspace which is spanned by the first 
M principal components of the data (Bourlard and Kamp, 1988; Baldi and Hornik, 
1989). Thus, the vectors of weights which lead into the hidden units in Figure 12.18 
form a basis set which spans the principal subspace. Note, however, that these vec- 
tors need not be orthogonal or normalized. This result is unsurprising, since both 
principal component analysis and the neural network are using linear dimensionality 
reduction and are minimizing the same sum-of-squares error function. 

It might be thought that the limitations of a linear dimensionality reduction could 
be overcome by using nonlinear (sigmoidal) activation functions for the hidden units 
in the network in Figure 12.18. However, even with nonlinear hidden units, the min- 
imum error solution is again given by the projection onto the principal component 
subspace (Bourlard and Kamp, 1988). There is therefore no advantage in using two- 
layer neural networks to perform dimensionality reduction. Standard techniques for 
principal component analysis (based on singular value decomposition) are guaran- 
teed to give the correct solution in finite time, and they also generate an ordered set 
of eigenvalues with corresponding orthonormal eigenvectors. 
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Figure 12.19 Addition of extra hidden lay- 
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ers of nonlinear units gives an 
autoassociative network which 
can perform a nonlinear dimen- 
sionality reduction. 


The situation is different, however, if additional hidden layers are permitted in 
the network. Consider the four-layer autoassociative network shown in Figure 12.19. 
Again the output units are linear, and the M units in the second hidden layer can also 
be linear, however, the first and third hidden layers have sigmoidal nonlinear activa- 
tion functions. The network is again trained by minimization of the error function 
(12.91). We can view this network as two successive functional mappings F; and 
F2, as indicated in Figure 12.19. The first mapping F; projects the original D- 
dimensional data onto an M-dimensional subspace S defined by the activations of 
the units in the second hidden layer. Because of the presence of the first hidden layer 
of nonlinear units, this mapping is very general, and in particular is not restricted to 
being linear. Similarly, the second half of the network defines an arbitrary functional 
mapping from the /-dimensional space back into the original D-dimensional input 
space. This has a simple geometrical interpretation, as indicated for the case D = 3 
and M = 2 in Figure 12.20. 

Such a network effectively performs a nonlinear principal component analysis. 


T3 


T2 


Figure 12.20 Geometrical interpretation of the mappings performed by the network in Figure 12.19 for the case 
of D = 3 inputs and M = 2 units in the middle hidden layer. The function F2 maps from an M-dimensional 
space S into a D-dimensional space and therefore defines the way in which the space S is embedded within the 
original x-space. Since the mapping F2 can be nonlinear, the embedding of S can be nonplanar, as indicated 
in the figure. The mapping F: then defines a projection of points in the original D-dimensional space into the 
M-dimensional subspace S. 
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It has the advantage of not being limited to linear transformations, although it con- 
tains standard principal component analysis as a special case. However, training 
the network now involves a nonlinear optimization problem, since the error function 
(12.91) is no longer a quadratic function of the network parameters. Computation- 
ally intensive nonlinear optimization techniques must be used, and there is the risk of 
finding a suboptimal local minimum of the error function. Also, the dimensionality 
of the subspace must be specified before training the network. 


12.4.3 Modelling nonlinear manifolds 


As we have already noted, many natural sources of data correspond to low- 
dimensional, possibly noisy, nonlinear manifolds embedded within the higher di- 
mensional observed data space. Capturing this property explicitly can lead to im- 
proved density modelling compared with more general methods. Here we consider 
briefly a range of techniques that attempt to do this. 

One way to model the nonlinear structure is through a combination of linear 
models, so that we make a piece-wise linear approximation to the manifold. This can 
be obtained, for instance, by using a clustering technique such as K-means based on 
Euclidean distance to partition the data set into local groups with standard PCA ap- 
plied to each group. A better approach is to use the reconstruction error for cluster 
assignment (Kambhatla and Leen, 1997; Hinton et al., 1997) as then a common cost 
function is being optimized in each stage. However, these approaches still suffer 
from limitations due to the absence of an overall density model. By using prob- 
abilistic PCA it is straightforward to define a fully probabilistic model simply by 
considering a mixture distribution in which the components are probabilistic PCA 
models (Tipping and Bishop, 1999a). Such a model has both discrete latent vari- 
ables, corresponding to the discrete mixture, as well as continuous latent variables, 
and the likelihood function can be maximized using the EM algorithm. A fully 
Bayesian treatment, based on variational inference (Bishop and Winn, 2000), allows 
the number of components in the mixture, as well as the effective dimensionalities 
of the individual models, to be inferred from the data. There are many variants of 
this model in which parameters such as the W matrix or the noise variances are tied 
across components in the mixture, or in which the isotropic noise distributions are 
replaced by diagonal ones, giving rise to a mixture of factor analysers (Ghahramani 
and Hinton, 1996a; Ghahramani and Beal, 2000). The mixture of probabilistic PCA 
models can also be extended hierarchically to produce an interactive data visualiza- 
tion algorithm (Bishop and Tipping, 1998). 

An alternative to considering a mixture of linear models is to consider a single 
nonlinear model. Recall that conventional PCA finds a linear subspace that passes 
close to the data in a least-squares sense. This concept can be extended to one- 
dimensional nonlinear surfaces in the form of principal curves (Hastie and Stuetzle, 
1989). We can describe a curve in a D-dimensional data space using a vector-valued 
function f(A}, which is a vector each of whose elements is a function of the scalar À. 
There are many possible ways to parameterize the curve, of which a natural choice 
is the arc length along the curve. For any given point X in data space, we can find 
the point on the curve that is closest in Euclidean distance. We denote this point by 
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A = ge(x) because it depends on the particular curve f(A). For a continuous data 
density p(x), a principal curve is defined as one for which every point on the curve 
is the mean of all those points in data space that project to it, so that 


E [x|ge(x) = A} = f(A). (12.92) 


For a given continuous density, there can be many principal curves. In practice, we 
are interested in finite data sets, and we also wish to restrict attention to smooth 
curves. Hastie and Stuetzle (1989) propose a two-stage iterative procedure for find- 
ing such principal curves, somewhat reminiscent of the EM algorithm for PCA. The 
curve is initialized using the first principal component, and then the algorithm alter- 
nates between a data projection step and curve re-estimation step. In the projection 
step, each data point is assigned to a value of À corresponding to the closest point 
on the curve. Then in the re-estimation step, each point on the curve is given by 
a weighted average of those points that project to nearby points on the curve, with 
points closest on the curve given the greatest weight. In the case where the subspace 
is constrained to be linear, the procedure converges to the first principal component 
and is equivalent to the power method for finding the largest eigenvector of the co- 
variance matrix. Principal curves can be generalized to multidimensional manifolds 
called principal surfaces although these have found limited use due to the difficulty 
of data smoothing in higher dimensions even for two-dimensional manifolds. 

PCA is often used to project a data set onto a lower-dimensional space, for ex- 
ample two dimensional, for the purposes of visualization. Another linear technique 
with a similar aim is multidimensional scaling, or MDS (Cox and Cox, 2000). It finds 
a low-dimensional projection of the data such as to preserve, as closely as possible, 
the pairwise distances between data points, and involves finding the eigenvectors of 
the distance matrix. In the case where the distances are Euclidean, it gives equivalent 
results to PCA. The MDS concept can be extended to a wide variety of data types 
specified in terms of a similarity matrix, giving nonmetric MDS. 

Two other nonprobabilistic methods for dimensionality reduction and data vi- 
sualization are worthy of mention. Locally linear embedding, or LLE (Roweis and 
Saul, 2000) first computes the set of coefficients that best reconstructs each data 
point from its neighbours. These coefficients are arranged to be invariant to rota- 
tions, translations, and scalings of that data point and its neighbours, and hence they 
characterize the local geometrical properties of the neighbourhood. LLE then maps 
the high-dimensional data points down to a lower dimensional space while preserv- 
ing these neighbourhood coefficients. If the local neighbourhood for a particular 
data point can be considered linear, then the transformation can be achieved using 
a combination of translation, rotation, and scaling, such as to preserve the angles 
formed between the data points and their neighbours. Because the weights are in- 
variant to these transformations, we expect the same weight values to reconstruct the 
data points in the low-dimensional space as in the high-dimensional data space. In 
spite of the nonlinearity, the optimization for LLE does not exhibit local minima. 

In isometric feature mapping, or isomap (Tenenbaum et al., 2000), the goal is 
to project the data to a lower-dimensional space using MDS, but where the dissim- 
ilarities are defined in terms of the geodesic distances measured along the mani- 
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fold. For instance, if two points lie on a circle, then the geodesic is the arc-length 
distance measured around the circumference of the circle not the straight line dis- 
tance measured along the chord connecting them. The algorithm first defines the 
neighbourhood for each data point, either by finding the K nearest neighbours or by 
finding all points within a sphere of radius e. A graph is then constructed by link- 
ing all neighbouring points and labelling them with their Euclidean distance. The 
geodesic distance between any pair of points is then approximated by the sum of 
the arc lengths along the shortest path connecting them (which itself is found using 
standard algorithms). Finally, metric MDS is applied to the geodesic distance matrix 
to find the low-dimensional projection. 

Our focus in this chapter has been on models for which the observed vari- 
ables are continuous. We can also consider models having continuous latent vari- 
ables together with discrete observed variables, giving rise to latent trait models 
(Bartholomew, 1987). In this case, the marginalization over the continuous latent 
variables, even for a linear relationship between latent and observed variables, can- 
not be performed analytically, and so more sophisticated techniques are required. 
Tipping (1999) uses variational inference in a model with a two-dimensional latent 
space, allowing a binary data set to be visualized analogously to the use of PCA to 
visualize continuous data. Note that this model is the dual of the Bayesian logistic 
regression problem discussed in Section 4.5. In the case of logistic regression we 
have N observations of the feature vector @,, which are parameterized by a single 
parameter vector w, whereas in the latent space visualization model there is a single 
latent space variable x (analogous to @) and N copies of the latent variable wp. A 
generalization of probabilistic latent variable models to general exponential family 
distributions is described in Collins et al. (2002). 

We have already noted that an arbitrary distribution can be formed by taking a 
Gaussian random variable and transforming it through a suitable nonlinearity. This 
is exploited in a general latent variable model called a density network (MacKay, 
1995; MacKay and Gibbs, 1999) in which the nonlinear function is governed by a 
multilayered neural network. If the network has enough hidden units, it can approx- 
imate a given nonlinear function to any desired accuracy. The downside of having 
such a flexible model is that the marginalization over the latent variables, required in 
order to obtain the likelihood function, is no longer analytically tractable. Instead, 
the likelihood is approximated using Monte Carlo techniques by drawing samples 
from the Gaussian prior. The marginalization over the latent variables then becomes 
a simple sum with one term for each sample. However, because a large number 
of sample points may be required in order to give an accurate representation of the 
marginal, this procedure can be computationally costly. 

If we consider more restricted forms for the nonlinear function, and make an ap- 
propriate choice of the latent variable distribution, then we can construct a latent vari- 
able model that is both nonlinear and efficient to train. The generative topographic 
mapping, or GTM (Bishop et al., 1996; Bishop et al., 1997a; Bishop et al., 1998b) 
uses a latent distribution that is defined by a finite regular grid of delta functions over 
the (typically two-dimensional) latent space. Marginalization over the latent space 
then simply involves summing over the contributions from each of the grid locations. 
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Figure 12.21 Plot of the oil flow data set visualized using PCA on the left and GTM on the right. For the GTM 
model, each data point is plotted at the mean of its posterior distribution in latent space. The nonlinearity of the 
GTM model allows the separation between the groups of data points to be seen more clearly. 
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The nonlinear mapping is given by a linear regression model that allows for general 
nonlinearity while being a linear function of the adaptive parameters. Note that the 
usual limitation of linear regression models arising from the curse of dimensionality 
does not arise in the context of the GTM since the manifold generally has two dimen- 
sions irrespective of the dimensionality of the data space. A consequence of these 
two choices is that the likelihood function can be expressed analytically in closed 
form and can be optimized efficiently using the EM algorithm. The resulting GTM 
model fits a two-dimensional nonlinear manifold to the data set, and by evaluating 
the posterior distribution over latent space for the data points, they can be projected 
back to the latent space for visualization purposes. Figure 12.21 shows a comparison 
of the oil data set visualized with linear PCA and with the nonlinear GTM, 

The GTM can be seen as a probabilistic version of an earlier model called the self 
organizing map, or SOM (Kohonen, 1982; Kohonen, 1995), which also represents 
a two-dimensional nonlinear manifold as a regular array of discrete points. The 
SOM is somewhat reminiscent of the K-means algorithm in that data points are 
assigned to nearby prototype vectors that are then subsequently updated. Initially, 
the prototypes are distributed at random, and during the training process they ‘self 
organize’ so as to approximate a smooth manifold. Unlike A-means, however, the 
SOM is not optimizing any well-defined cost function (Erwin et al., 1992) making it 
difficult to set the parameters of the model and to assess convergence. There is also 
no guarantee that the ‘self-organization’ will take place as this is dependent on the 
choice of appropriate parameter values for any particular data set. 

By contrast, GTM optimizes the log likelihood function, and the resulting model 
defines a probability density in data space. In fact, it corresponds to a constrained 
mixture of Gaussians in which the components share a common variance, and the 
means are constrained to lie on a smooth two-dimensional manifold. This proba- 
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bilistic foundation also makes it very straightforward to define generalizations of 
GTM (Bishop et al., 1998a) such as a Bayesian treatment, dealing with missing val- 
ues, a principled extension to discrete variables, the use of Gaussian processes to 
define the manifold, or a hierarchical GTM model (Tino and Nabney, 2002). 

Because the manifold in GTM is defined as a continuous surface, not just at the 
prototype vectors as in the SOM, it is possible to compute the magnification factors 
corresponding to the local expansions and compressions of the manifold needed to 
fit the data set (Bishop et al., 1997b) as well as the directional curvatures of the 
manifold (Tino et al., 2001). These can be visualized along with the projected data 
and provide additional insight into the model. 


Exercises 


Appendix E 


12.1 


12.2 


œx) ENDA In this exercise, we use proof by induction to show that the linear 
projection onto an M-dimensional subspace that maximizes the variance of the pro- 
jected data is defined by the M eigenvectors of the data covariance matrix S, given 
by (12.3), corresponding to the M largest eigenvalues. In Section 12.1, this result 
was proven for the case of M = 1. Now suppose the result holds for some general 
value of M and show that it consequently holds for dimensionality M + 1. To do 
this, first set the derivative of the variance of the projected data with respect to a 
vector uļm+ı defining the new direction in data space equal to zero. This should 
be done subject to the constraints that um}: be orthogonal to the existing vectors 
ui,..., Um, and also that it be normalized to unit length. Use Lagrange multipli- 
ers to enforce these constraints. Then make use of the orthonormality properties of 
the vectors u;,..., Um to show that the new vector um+1 is an eigenvector of S. 
Finally, show that the variance is maximized if the eigenvector is chosen to be the 
one corresponding to eigenvector \,7+1 where the eigenvalues have been ordered in 
decreasing value. 


(xx) Show that the minimum value of the PCA distortion measure J given by 
(12.15) with respect to the u;, subject to the orthonormality constraints (12.7), is 
obtained when the u; are eigenvectors of the data covariance matrix S. To do this, 
introduce a matrix H of Lagrange multipliers, one for each constraint, so that the 
modified distortion measure, in matrix notation reads 


J=Tr {ost} +Tr {Ha = 070) (12.93) 


where U is a matrix of dimension D x (D — M) whose columns are given by uj. 
Now minimize J with respect to Ü and show that the solution satisfies SU = OH. 
Clearly, one possible solution is that the columns of U are eigenvectors of S, in 
which case H is a diagonal matrix containing the corresponding eigenvalues. To 
obtain the general solution, show that H can be assumed to be a symmetric matrix, 


and by using its eigenvector expansion show that the general solution to SŪ = UH 
gives the same value for J as the specific solution in which the columns of U are 
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12.3 


12.4 


12.5 


12.6 


12.7 


12.8 


12.9 


12.10 


12.11 


12.12 


12.13 


the eigenvectors of S. Because these solutions are all equivalent, it is convenient to 
choose the eigenvector solution. 


(x) Verify that the eigenvectors defined by (12.30) are normalized to unit length, 
assuming that the eigenvectors v; have unit length. 


(x) | www | Suppose we replace the zero-mean, unit-covariance latent space distri- 
bution (12.31) in the probabilistic PCA model by a general Gaussian distribution of 
the form N (z|m, £). By redefining the parameters of the model, show that this leads 
to an identical model for the marginal distribution p(x) over the observed variables 
for any valid choice of m and &. 


(xx) Let x be a D-dimensional random variable having a Gaussian distribution 
given by N(x|u, ©), and consider the M-dimensional random variable given by 
y = Ax + b where A is an M x D matrix. Show that y also has a Gaussian 
distribution, and find expressions for its mean and covariance. Discuss the form of 
this Gaussian distribution for M < D, for M = D, and for M > D. 


(x) FQ Draw a directed probabilistic graph for the probabilistic PCA model 
described in Section 12.2 in which the components of the observed variable x are 
shown explicitly as separate nodes. Hence verify that the probabilistic PCA model 
has the same independence structure as the naive Bayes model discussed in Sec- 
tion 8.2.2. 


(xx) By making use of the results (2.270) and (2.271) for the mean and covariance 
of a general distribution, derive the result (12.35) for the marginal distribution p(x) 
in the probabilistic PCA model. 


(x *) EY By making use of the result (2.116), show that the posterior distribution 
p(z|x) for the probabilistic PCA model is given by (12.42). 


(x) Verify that maximizing the log likelihood (12.43) for the probabilistic PCA 
model with respect to the parameter p gives the result pmr, = X where X is the 
mean of the data vectors. 


(xx) By evaluating the second derivatives of the log likelihood function (12.43) for 
the probabilistic PCA model with respect to the parameter u, show that the stationary 
point uy; = X represents the unique maximum. 


(x x) RAMA Show that in the limit o? — 0, the posterior mean for the probabilistic 
PCA model becomes an orthogonal projection onto the principal subspace, as in 
conventional PCA. 


(xx) For o? > 0 show that the posterior mean in the probabilistic PCA model is 
shifted towards the origin relative to the orthogonal projection. 


(xx) Show that the optimal reconstruction of a data point under probabilistic PCA, 
according to the least squares projection cost of conventional PCA, is given by 


X = Wm (W in Wat)’ ME[zZ|x]. (12.94) 


12.14 


12.15 


12.16 


12.17 


12.18 


12.19 


12.20 
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(x) The number of independent parameters in the covariance matrix for the proba- 
bilistic PCA model with an A4-dimensional latent space and a D-dimensional data 
space is given by (12.51). Verify that in the case of M = D — 1, the number of 
independent parameters is the same as in a general covariance Gaussian, whereas for 
M = Q itis the same as for a Gaussian with an isotropic covariance. 


(x *) FRY Derive the M-step equations (12.56) and (12.57) for the probabilistic 
PCA model by maximization of the expected complete-data log likelihood function 
given by (12.53). 


(xxx) In Figure 12.11, we showed an application of probabilistic PCA to a data set 
in which some of the data values were missing at random. Derive the EM algorithm 
for maximizing the likelihood function for the probabilistic PCA model in this situ- 
ation. Note that the {z,,}, as well as the missing data values that are components of 
the vectors {x,,}, are now latent variables. Show that in the special case in which all 
of the data values are observed, this reduces to the EM algorithm for probabilistic 
PCA derived in Section 12.2.2. 


(x *) EATA Let W be a D x M matrix whose columns define a linear subspace 
of dimensionality M embedded within a data space of dimensionality D, and let y 
be a D-dimensional vector. Given a data set {x,} where n = 1,..., N, we can 
approximate the data points using a linear mapping from a set of /-dimensional 
vectors {Zn}, so that x, is approximated by Wz,, + ys. The associated sum-of- 
squares reconstruction cost is given by 


N 
J =X |x ~~ Waal? (12.95) 


n=l 


First show that minimizing J with respect to yz leads to an analogous expression with 
Xn and Z, replaced by zero-mean variables x,, — X and z,, — Z, respectively, where X 
and Z denote sample means. Then show that minimizing J with respect to Zn, where 
W is kept fixed, gives rise to the PCA E step (12.58), and that minimizing J with 
respect to W, where {z,,} is kept fixed, gives rise to the PCA M step (12.59). 


(x) Derive an expression for the number of independent parameters in the factor 
analysis model described in Section 12.2.4. 


œx) FQ Show that the factor analysis model described in Section 12.2.4 is 
invariant under rotations of the latent space coordinates. 


(xx) By considering second derivatives, show that the only stationary point of 
the log likelihood function for the factor analysis model discussed in Section 12.2.4 
with respect to the parameter yz is given by the sample mean defined by (12.1). 
Furthermore, show that this stationary point is a maximum. 


(xx) Derive the formulae (12.66) and (12.67) for the E step of the EM algorithm 
for factor analysis. Note that from the result of Exercise 12.20, the parameter p can 
be replaced by the sample mean X. 
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12.22 


12.23 


12.24 


12.25 


12.26 


12.27 


12.28 


(«*) Write down an expression for the expected complete-data log likelihood func- 
tion for the factor analysis model, and hence derive the corresponding M step equa- 
tions (12.69) and (12.70). 


(x) FQ Draw a directed probabilistic graphical model representing a discrete 
mixture of probabilistic PCA models in which each PCA model has its own values 
of W, u, and o?. Now draw a modified graph in which these parameter values are 
shared between the components of the mixture. 


(xxx) We saw in Section 2.3.7 that Student’s t-distribution can be viewed as an 
infinite mixture of Gaussians in which we marginalize with respect to a continu- 
ous latent variable. By exploiting this representation, formulate an EM algorithm 
for maximizing the log likelihood function for a multivariate Student’s t-distribution 
given an observed set of data points, and derive the forms of the E and M step equa- 
tions. 


(x>) www | Consider a linear-Gaussian latent-variable model having a latent space 
distribution p(z) = M (x|0, I) and a conditional distribution for the observed vari- 
able p(x|z) = N(x|Wz + u, ®) where ® is an arbitrary symmetric, positive- 
definite noise covariance matrix. Now suppose that we make a nonsingular linear 
transformation of the data variables x — Ax, where A isa D x D matrix. If 
Myr» Wm and ®yz represent the maximum likelihood solution corresponding to 
the original untransformed data, show that A py, AW mu, and A®ymL A" will rep- 
resent the corresponding maximum likelihood solution for the transformed data set. 
Finally, show that the form of the model is preserved in two cases: (i) A is a diagonal 
matrix and ® is a diagonal matrix. This corresponds to the case of factor analysis. 
The transformed ® remains diagonal, and hence factor analysis is covariant under 
component-wise re-scaling of the data variables; (ii) A is orthogonal and ® is pro- 
portional to the unit matrix so that ® = g°I. This corresponds to probabilistic PCA. 
The transformed ® matrix remains proportional to the unit matrix, and hence proba- 
bilistic PCA is covariant under a rotation of the axes of data space, as is the case for 
conventional PCA. 


(xx) Show that any vector a; that satisfies (12.80) will also satisfy (12.79). Also, 
show that for any solution of (12.80) having eigenvalue A, we can add any multiple 
of an eigenvector of K having zero eigenvalue, and obtain a solution to (12.79) 
that also has eigenvalue A. Finally, show that such modifications do not affect the 
principal-component projection given by (12.82). 


(«*) Show that the conventional linear PCA algorithm is recovered as a special case 
Taf 


of kernel PCA if we choose the linear kernel function given by k(x, x’) = xx’. 
œx) ENTI Use the transformation property (1.27) of a probability density under 
a change of variable to show that any density p(y) can be obtained from a fixed 
density g(x) that is everywhere nonzero by making a nonlinear change of variable 
y = f(x) in which f(x) is a monotonic function so that 0 < f'(x) < co. Write 
down the differential equation satisfied by f(x) and draw a diagram illustrating the 
transformation of the density. 
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12.29 (x) EY Suppose that two variables z; and z3 are independent so that p(z1, 22) = 
p(21)p(z2). Show that the covariance matrix between these variables is diagonal. 
This shows that independence is a sufficient condition for two variables to be un- 
correlated. Now consider two variables yı and ye in which —1 < yı < 1 and 
y2 = y2. Write down the conditional distribution p(y2|y,) and observe that this is 
dependent on yı, showing that the two variables are not independent. Now show 
that the covariance matrix between these two variables is again diagonal. To do this, 
use the relation p(y, y2) = p(yi)p(ye|y1) to show that the off-diagonal terms are 
zero. This counter-example shows that zero correlation is not a sufficient condition 
for independence. 


13 


So far in this book, we have focussed primarily on sets of data points that were as- 
sumed to be independent and identically distributed (i.i.d.). This assumption allowed 
us to express the likelihood function as the product over all data points of the prob- 
ability distribution evaluated at each data point. For many applications, however, 
the i.i.d. assumption will be a poor one. Here we consider a particularly important 
class of such data sets, namely those that describe sequential data. These often arise 
through measurement of time series, for example the rainfall measurements on suc- 
cessive days at a particular location, or the daily values of a currency exchange rate, 
or the acoustic features at successive time frames used for speech recognition. An 
example involving speech data is shown in Figure 13.1. Sequential data can also 
arise in contexts other than time series, for example the sequence of nucleotide base 
pairs along a strand of DNA or the sequence of characters in an English sentence. 
For convenience, we shall sometimes refer to ‘past’ and ‘future’ observations in a 
sequence. However, the models explored in this chapter are equally applicable to all 
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Figure 13.1 


Example of a spectro- 
gram of the spoken words “Bayes’ theo- 10000 
rem” showing a plot of the intensity of the 
spectral coefficients versus time index. 
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forms of sequential data, not just temporal sequences. 

It is useful to distinguish between stationary and nonstationary sequential dis- 
tributions. In the stationary case, the data evolves in time, but the distribution from 
which it is generated remains the same. For the more complex nonstationary situa- 
tion, the generative distribution itself is evolving with time. Here we shall focus on 
the stationary case. 

For many applications, such as financial forecasting, we wish to be able to pre- 
dict the next value in a time series given observations of the previous values. In- 
tuitively, we expect that recent observations are likely to be more informative than 
more historical observations in predicting future values. The example in Figure 13.1 
shows that successive observations of the speech spectrum are indeed highly cor- 
related. Furthermore, it would be impractical to consider a general dependence of 
future observations on all previous observations because the complexity of such a 
model would grow without limit as the number of observations increases. This leads 
us to consider Markov models in which we assume that future predictions are inde- 
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The simplest approach to 
modelling a sequence of ob- X LO O 
servations is to treat them *! 2 3 X4 


as independent, correspond- 
ing to a graph without links. 


pendent of all but the most recent observations. 

Although such models are tractable, they are also severely limited. We can ob- 
tain a more general framework, while still retaining tractability, by the introduction 
of latent variables, leading to state space models. As in Chapters 9 and 12, we shall 
see that complex models can thereby be constructed from simpler components (in 
particular, from distributions belonging to the exponential family) and can be read- 
ily characterized using the framework of probabilistic graphical models. Here we 
focus on the two most important examples of state space models, namely the hid- 
den Markov model, in which the latent variables are discrete, and linear dynamical 
systems, in which the latent variables are Gaussian. Both models are described by di- 
rected graphs having a tree structure (no loops) for which inference can be performed 
efficiently using the sum-product algorithm. 


Markov Models 


The easiest way to treat sequential data would be simply to ignore the sequential 
aspects and treat the observations as i.i.d., corresponding to the graph in Figure 13.2. 
Such an approach, however, would fail to exploit the sequential patterns in the data, 
such as correlations between observations that are close in the sequence. Suppose, 
for instance, that we observe a binary variable denoting whether on a particular day 
it rained or not. Given a time series of recent observations of this variable, we wish 
to predict whether it will rain on the next day. If we treat the data as i.i.d., then the 
only information we can glean from the data is the relative frequency of rainy days. 
However, we know in practice that the weather often exhibits trends that may last for 
several days. Observing whether or not it rains today is therefore of significant help 
in predicting if it will rain tomorrow. 

To express such effects in a probabilistic model, we need to relax the i.i.d. as- 
sumption, and one of the simplest ways to do this is to consider a Markov model. 
First of all we note that, without loss of generality, we can use the product rule to 
express the joint distribution for a sequence of observations in the form 


N 
p(X,- --;, XN) = WE AT ee (13.1) 
n=1 


If we now assume that each of the conditional distributions on the right-hand side 
is independent of all previous observations except the most recent, we obtain the 
first-order Markov chain, which is depicted as a graphical model in Figure 13.3. The 
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Figure 13.3 
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Figure 13.4 


A first-order Markov chain of ob- 

servations {xn } in which the dis- ()}—C }-—C._ +. )— 
tribution p(xn|xn—1) of a particu- *1 = X3 X4 

lar observation x,, is conditioned 

on the value of the previous ob- 


servation xn-1. 
joint distribution for a sequence of N observations under this model is given by 


N 
P04, XN) = p(xı) II P(Xn|Xn-1). (13.2) 
n=2 


From the d-separation property, we see that the conditional distribution for observa- 
tion x,,, given all of the observations up to time n, is given by 


POG, iu. Xn) = PSA Mena) (13.3) 


which is easily verified by direct evaluation starting from (13.2) and using the prod- 
uct rule of probability. Thus if we use such a model to predict the next observation 
in a sequence, the distribution of predictions will depend only on the value of the im- 
mediately preceding observation and will be independent of all earlier observations. 

In most applications of such models, the conditional distributions p(x,,|Xn—1) 
that define the model will be constrained to be equal, corresponding to the assump- 
tion of a stationary time series. The model is then known as a homogeneous Markov 
chain. For instance, if the conditional distributions depend on adjustable parameters 
(whose values might be inferred from a set of training data), then all of the condi- 
tional distributions in the chain will share the same values of those parameters. 

Although this is more general than the independence model, it is still very re- 
strictive. For many sequential observations, we anticipate that the trends in the data 
over several successive observations will provide important information in predict- 
ing the next value. One way to allow earlier observations to have an influence is to 
move to higher-order Markov chains. If we allow the predictions to depend also on 
the previous-but-one value, we obtain a second-order Markov chain, represented by 
the graph in Figure 13.4. The joint distribution is now given by 


N 
p(X1,---,Xn) = p(&1)p(X2|X1) J [pnnt na): (13.4) 


n=3 


Again, using d-separation or by direct evaluation, we see that the conditional distri- 
bution of x, given x,,_, and x,,_»2 is independent of all observations x,,...X,_3. 


A second-order Markov chain, in 
which the conditional distribution 

of a particular observation x, 
depends on the values of the two 
previous observations x,_1 and xi 
Xn-2- 


X2 X3 X4 
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Figure 13.5 We can represent sequen- Zy Z2 Zn—1 Zn Zn+1 
tial data using a Markov chain of latent 

variables, with each observation condi- 

tioned on the state of the corresponding 

latent variable. This important graphical 

structure forms the foundation both for the Xı X2 
hidden Markov model and for linear dy- 

namical systems. 


Each observation is now influenced by two previous observations. We can similarly 
consider extensions to an M*® order Markov chain in which the conditional distri- 
bution for a particular variable depends on the previous M variables. However, we 
have paid a price for this increased flexibility because the number of parameters in 
the model is now much larger. Suppose the observations are discrete variables hav- 
ing K states. Then the conditional distribution p(x,,|x,,_1) in a first-order Markov 
chain will be specified by a set of K — 1 parameters for each of the K states of x,,_1 
giving a total of K(K — 1) parameters. Now suppose we extend the model to an 
M* order Markov chain, so that the joint distribution is built up from conditionals 
P(Xn|Xn—M,---,Xn—1). If the variables are discrete, and if the conditional distri- 
butions are represented by general conditional probability tables, then the number 
of parameters in such a model will have K“~1(K — 1) parameters. Because this 
grows exponentially with M, it will often render this approach impractical for larger 
values of M. 

For continuous variables, we can use linear-Gaussian conditional distributions 
in which each node has a Gaussian distribution whose mean is a linear function 
of its parents. This is known as an autoregressive or AR model (Box et al., 1994; 
Thiesson et al., 2004). An alternative approach is to use a parametric model for 
P(Xn|Xn—M,--+-,Xn—1) such as a neural network. This technique is sometimes 
called a tapped delay line because it corresponds to storing (delaying) the previous 
M values of the observed variable in order to predict the next value. The number 
of parameters can then be much smaller than in a completely general model (for ex- 
ample it may grow linearly with M), although this is achieved at the expense of a 
restricted family of conditional distributions. 

Suppose we wish to build a model for sequences that is not limited by the 
Markov assumption to any order and yet that can be specified using a limited number 
of free parameters. We can achieve this by introducing additional latent variables to 
permit a rich class of models to be constructed out of simple components, as we did 
with mixture distributions in Chapter 9 and with continuous latent variable models in 
Chapter 12. For each observation x,,, we introduce a corresponding latent variable 
Zn (which may be of different type or dimensionality to the observed variable). We 
now assume that it is the latent variables that form a Markov chain, giving rise to the 
graphical structure known as a state space model, which is shown in Figure 13.5. It 
satisfies the key conditional independence property that z,,_; and z,,,1 are indepen- 
dent given Zn, so that 

Zn+1 LL Zn-1 | Zn. (13.5) 
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The joint distribution for this model is given by 


N N 
P(X1 XN; Z1,- :, ZN) = p(z1) TT tens) [[ eGnlen). 03.6) 
n=2 m=i 


Using the d-separation criterion, we see that there is always a path connecting any 
two observed variables x,, and Xm via the latent variables, and that this path is never 
blocked. Thus the predictive distribution p(X,41|X1,...,Xn) for observation Xn+1 
given all previous observations does not exhibit any conditional independence prop- 
erties, and so our predictions for x,+4; depends on all previous observations. The 
observed variables, however, do not satisfy the Markov property at any order. We 
shall discuss how to evaluate the predictive distribution in later sections of this chap- 
ter. 

There are two important models for sequential data that are described by this 
graph. If the latent variables are discrete, then we obtain the hidden Markov model, 
or HMM (Elliott et al., 1995). Note that the observed variables in an HMM may 
be discrete or continuous, and a variety of different conditional distributions can be 
used to model them. If both the latent and the observed variables are Gaussian (with 
a linear-Gaussian dependence of the conditional distributions on their parents), then 
we obtain the linear dynamical system. 


Hidden Markov Models 


The hidden Markov model can be viewed as a specific instance of the state space 
model of Figure 13.5 in which the latent variables are discrete. However, if we 
examine a single time slice of the model, we see that it corresponds to a mixture 
distribution, with component densities given by p(x|z). It can therefore also be 
interpreted as an extension of a mixture model in which the choice of mixture com- 
ponent for each observation is not selected independently but depends on the choice 
of component for the previous observation. The HMM is widely used in speech 
recognition (Jelinek, 1997; Rabiner and Juang, 1993), natural language modelling 
(Manning and Schütze, 1999), on-line handwriting recognition (Nag et al., 1986), 
and for the analysis of biological sequences such as proteins and DNA (Krogh et al., 
1994; Durbin et al., 1998; Baldi and Brunak, 2001). 

As in the case of a standard mixture model, the latent variables are the discrete 
multinomial variables z,, describing which component of the mixture is responsible 
for generating the corresponding observation Xn. Again, it is convenient to use a 
1-of-k coding scheme, as used for mixture models in Chapter 9. We now allow the 
probability distribution of Zn to depend on the state of the previous latent variable 
Zņ„—1 through a conditional distribution p(z,,|Z,,_1). Because the latent variables are 
K-dimensional binary variables, this conditional distribution corresponds to a table 
of numbers that we denote by A, the elements of which are known as transition 
probabilities. They are given by Ajk = p(Znk = 1|Zn-1,; = 1), and because they 
are probabilities, they satisfy 0 < Aj, < 1 with ` ; Ajk = 1, so that the matrix A 
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Figure 13.6 Transition diagram showing a model whose la- A22 

tent variables have three possible states corre- 

sponding to the three boxes. The black lines 

denote the elements of the transition matrix At 

Ajk. 

= Ai» 
k=2 
A32 | | A23 ka O >e 

k= 
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has K (K -—1) independent parameters. We can then write the conditional distribution 
explicitly in the form 


P(Zn|Zn—1,A) = WNE aa (13.7) 


The initial latent node zı is special in that it does not have a parent node, and so 
it has a marginal distribution p(z,) represented by a vector of probabilities m with 
elements Tk = p(21x = 1), so that 


K 
p(z, |r) = II m (13.8) 
k=1 


where X`, 7% = 1. 

The transition matrix is sometimes illustrated diagrammatically by drawing the 
states as nodes in a state transition diagram as shown in Figure 13.6 for the case of 
K = 3. Note that this does not represent a probabilistic graphical model, because 
the nodes are not separate variables but rather states of a single variable, and so we 
have shown the states as boxes rather than circles. 

It is sometimes useful to take a state transition diagram, of the kind shown in 
Figure 13.6, and unfold it over time. This gives an alternative representation of the 
transitions between latent states, known as a lattice or trellis diagram, and which is 
shown for the case of the hidden Markov model in Figure 13.7. 

The specification of the probabilistic model is completed by defining the con- 
ditional distributions of the observed variables p(Xn|Zn, @), where @ is a set of pa- 
rameters governing the distribution. These are known as emission probabilities, and 
might for example be given by Gaussians of the form (9.11) if the elements of x are 
continuous variables, or by conditional probability tables if x is discrete. Because 
Xn is observed, the distribution p(xXn|Zn, @) consists, for a given value of ¢, of a 
vector of K numbers corresponding to the K possible states of the binary vector Zn. 
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Figure 13.7 If we unfold the state transition dia- 
gram of Figure 13.6 over time, we obtain a lattice, 
or trellis, representation of the latent states. Each 
column of this diagram corresponds to one of the 
latent variables zn. 


We can represent the emission probabilities in the form 


K 
P(&nlzn, 6) = | [ 2ænlo) e. (13.9) 
k=1 


We shall focuss attention on homogeneous models for which all of the condi- 
tional distributions governing the latent variables share the same parameters A, and 
similarly all of the emission distributions share the same parameters @ (the extension 
to more general cases is straightforward). Note that a mixture model for an 1.i.d. data 
set corresponds to the special case in which the parameters Ajy are the same for all 
values of j, so that the conditional distribution p(z,,|Z,_1) is independent of z,,_1. 
This corresponds to deleting the horizontal links in the graphical model shown in 
Figure 13.5. 

The joint probability distribution over both latent and observed variables is then 
given by 


N N 
p(X, Z|8) = p(zı|7) TI P(ZnlZn-1, a) [Dnm p) 0310) 


m=1 


where X = {x1,..., XN}, Z = {z1,...,Zn}, and 0 = {7, A, ġ} denotes the set 
of parameters governing the model. Most of our discussion of the hidden Markov 
model will be independent of the particular choice of the emission probabilities. 
Indeed, the model is tractable for a wide range of emission distributions including 
discrete tables, Gaussians, and mixtures of Gaussians. It is also possible to exploit 
Exercise 13.4 discriminative models such as neural networks. These can be used to model the 
emission density p(x|z) directly, or to provide a representation for p(z|x) that can 
be converted into the required emission density p(x|z) using Bayes’ theorem (Bishop 
et al., 2004). 
We can gain a better understanding of the hidden Markov model by considering 
it from a generative point of view. Recall that to generate samples from a mixture of 
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Figure 13.8 Illustration of sampling from a hidden Markov model having a 3-state latent variable z and a 
Gaussian emission model p(x|z) where x is 2-dimensional. (a) Contours of constant probability density for the 
emission distributions corresponding to each of the three states of the latent variable. (b) A sample of 50 points 
drawn from the hidden Markov model, colour coded according to the component that generated them and with 
lines connecting the successive observations. Here the transition matrix was fixed so that in any state there is a 
5% probability of making a transition to each of the other states, and consequently a 90% probability of remaining 


in the same state. 
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Gaussians, we first chose one of the components at random with probability given by 
the mixing coefficients 7; and then generate a sample vector x from the correspond- 
ing Gaussian component. This process is repeated N times to generate a data set of 
N independent samples. In the case of the hidden Markov model, this procedure is 
modified as follows. We first choose the initial latent variable zı with probabilities 
governed by the parameters mą and then sample the corresponding observation x. 
Now we choose the state of the variable z2 according to the transition probabilities 
p(Z2|Z1) using the already instantiated value of zı. Thus suppose that the sample for 
zı corresponds to state 7. Then we choose the state k of Z with probabilities Aj, 
fork = 1,..., K. Once we know zz we can draw a sample for x> and also sample 
the next latent variable z3 and so on. This is an example of ancestral sampling for 
a directed graphical model. If, for instance, we have a model in which the diago- 
nal transition elements A;; are much larger than the off-diagonal elements, then a 
typical data sequence will have long runs of points generated from a single compo- 
nent, with infrequent transitions from one component to another. The generation of 
samples from a hidden Markov model is illustrated in Figure 13.8. 

There are many variants of the standard HMM model, obtained for instance by 
imposing constraints on the form of the transition matrix A (Rabiner, 1989). Here we 
mention one of particular practical importance called the left-to-right HMM, which 
is obtained by setting the elements A;;, of A to zero if k < j, as illustrated in the 
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Figure 13.10 Lattice diagram for a 3-state left- 
to-right HMM in which the state index k is allowed 
to increase by at most 1 at each transition. 
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Figure 13.9 Example of the state transition diagram for a 3-state Aj, Ago A33 


left-to-right hidden Markov model. Note that once a O 


state has been vacated, it cannot later be re-entered. O O 
O00 
ae ——— 


Aj3 


state transition diagram for a 3-state HMM in Figure 13.9. Typically for such models 
the initial state probabilities for p(z; ) are modified so that p(z11) = 1 and p(z;) = 0 
for j # 1, in other words every sequence is constrained to start in state 7 = 1. The 
transition matrix may be further constrained to ensure that large changes in the state 
index do not occur, so that Aj, = 0 if k > j + A. This type of model is illustrated 
using a lattice diagram in Figure 13.10. 

Many applications of hidden Markov models, for example speech recognition, 
or on-line character recognition, make use of left-to-right architectures. As an illus- 
tration of the left-to-right hidden Markov model, we consider an example involving 
handwritten digits. This uses on-line data, meaning that each digit is represented 
by the trajectory of the pen as a function of time in the form of a sequence of pen 
coordinates, in contrast to the off-line digits data, discussed in Appendix A, which 
comprises static two-dimensional pixellated images of the ink. Examples of the on- 
line digits are shown in Figure 13.11. Here we train a hidden Markov model on a 
subset of data comprising 45 examples of the digit ‘2’. There are K = 16 states, 
each of which can generate a line segment of fixed length having one of 16 possible 
angles, and so the emission distribution is simply a 16 x 16 table of probabilities 
associated with the allowed angle values for each state index value. Transition prob- 
abilities are all set to zero except for those that keep the state index k the same or 
that increment it by 1, and the model parameters are optimized using 25 iterations of 
EM. We can gain some insight into the resulting model by running it generatively, as 
shown in Figure 13.11. 
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Figure 13.11 Top row: examples of on-line handwritten 


digits. Bottom row: synthetic digits sam- 
pled generatively from a left-to-right hid- 
den Markov model that has been trained 


on a data set of 45 handwritten digits. 


mm || 2) 2 


One of the most powerful properties of hidden Markov models is their ability to 
exhibit some degree of invariance to local warping (compression and stretching) of 
the time axis. To understand this, consider the way in which the digit ‘2’ is written 
in the on-line handwritten digits example. A typical digit comprises two distinct 
sections joined at a cusp. The first part of the digit, which starts at the top left, has a 
sweeping arc down to the cusp or loop at the bottom left, followed by a second more- 
or-less straight sweep ending at the bottom right. Natural variations in writing style 
will cause the relative sizes of the two sections to vary, and hence the location of the 
cusp or loop within the temporal sequence will vary. From a generative perspective 
such variations can be accommodated by the hidden Markov model through changes 
in the number of transitions to the same state versus the number of transitions to the 
successive state. Note, however, that if a digit ‘2’ is written in the reverse order, that 
is, Starting at the bottom right and ending at the top left, then even though the pen tip 
coordinates may be identical to an example from the training set, the probability of 
the observations under the model will be extremely small. In the speech recognition 
context, warping of the time axis is associated with natural variations in the speed of 
speech, and again the hidden Markov model can accommodate such a distortion and 
not penalize it too heavily. 


13.2.1 Maximum likelihood for the HMM 


If we have observed a data set X = {x1,..., Xy }, we can determine the param- 
eters of an HMM using maximum likelihood. The likelihood function is obtained 
from the joint distribution (13.10) by marginalizing over the latent variables 


p(X|0) = X` p(X, Z|). (13.11) 
Z 


Because the joint distribution p(X, Z|@) does not factorize over n (in contrast to the 
mixture distribution considered in Chapter 9), we cannot simply treat each of the 
summations over z,, independently. Nor can we perform the summations explicitly 
because there are N variables to be summed over, each of which has K states, re- 
sulting in a total of K™ terms. Thus the number of terms in the summation grows 
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exponentially with the length of the chain. In fact, the summation in (13.11) cor- 
responds to summing over exponentially many paths through the lattice diagram in 
Figure 13.7. 

We have already encountered a similar difficulty when we considered the infer- 
ence problem for the simple chain of variables in Figure 8.32. There we were able 
to make use of the conditional independence properties of the graph to re-order the 
summations in order to obtain an algorithm whose cost scales linearly, instead of 
exponentially, with the length of the chain. We shall apply a similar technique to the 
hidden Markov model. 

A further difficulty with the expression (13.11) for the likelihood function is that, 
because it corresponds to a generalization of a mixture distribution, it represents a 
summation over the emission models for different settings of the latent variables. 
Direct maximization of the likelihood function will therefore lead to complex ex- 
pressions with no closed-form solutions, as was the case for simple mixture models 
(recall that a mixture model for i.i.d. data is a special case of the HMM). 

We therefore turn to the expectation maximization algorithm to find an efficient 
framework for maximizing the likelihood function in hidden Markov models. The 
EM algorithm starts with some initial selection for the model parameters, which we 
denote by 6°". In the E step, we take these parameter values and find the posterior 
distribution of the latent variables p(Z|X, 0°"). We then use this posterior distri- 
bution to evaluate the expectation of the logarithm of the complete-data likelihood 
function, as a function of the parameters @, to give the function Q(0, 0°") defined 
by 

Q(0,0°") = X` p(Z|X, 0°") In p(X, Z|). (13.12) 
Z 


At this point, it is convenient to introduce some notation. We shall use 7(z,,) to 
denote the marginal posterior distribution of a latent variable z,,, and €(Z,_1, Zn) to 
denote the joint posterior distribution of two successive latent variables, so that 


(Zn) = PlZn|X, 0%) (13.13) 
ElZn=1;Z2n) = p(Zn—1,2n|X, 0"). (13.14) 


For each value of n, we can store 7(z,,) using a set of K nonnegative numbers 
that sum to unity, and similarly we can store €(Z,_1,Z,) using a K x K matrix of 
nonnegative numbers that again sum to unity. We shall also use y(znx) to denote the 
conditional probability of zn = 1, with a similar use of notation for €(2n—1,;, Znk) 
and for other probabilistic variables introduced later. Because the expectation of a 
binary random variable is just the probability that it takes the value 1, we have 


Zak) Elena] =e Y(Z)Znk (13.15) 


E(Zn—1,5) 2nk) 


E[Zn—1,3 nk] = 212) Z)zn—1,j2nk- (13.16) 


If we substitute the joint distribution p(X, Z|@) given by (13.10) into (13.12), 
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and make use of the definitions of y and £ , we obtain 


N K K 


Q(0, 0, 0°) = Yv (zik) Jnr +Y YO JE (Zn-1,j; Znk) In Age 


n=2 j=1 k=1 
N K 
p22. (zn) In p(Xn|b,)- (13.17) 
n=1 k=1 


The goal of the E step will be to evaluate the quantities 7(z,,) and €(Zp_1, Zn) effi- 
ciently, and we shall discuss this in detail shortly. 

In the M step, we maximize Q(6,0°") with respect to the parameters 9 = 
{7, A, o} in which we treat (Zn) and €(Zn—1, Zn) as constant. Maximization with 
respect to m and A is easily achieved using appropriate Lagrange multipliers with 
the results 


a = mis (13.18) 


do 14) 


N 


sce Zak) 


Ajk = a (13.19) 


Dod Eleni nt) 


l=1 n=2 


The EM algorithm must be initialized by choosing starting values for m and A, which 
should of course respect the summation constraints associated with their probabilis- 
tic interpretation. Note that any elements of m or A that are set to zero initially will 
remain zero in subsequent EM updates. A typical initialization procedure would 
involve selecting random starting values for these parameters subject to the summa- 
tion and non-negativity constraints. Note that no particular modification to the EM 
results are required for the case of left-to-right models beyond choosing initial values 
for the elements Ajg in which the appropriate elements are set to zero, because these 
will remain zero throughout. 

To maximize Q(0, 0°") with respect to @,, we notice that only the final term 
in (13.17) depends on @;, and furthermore this term has exactly the same form as 
the data-dependent term in the corresponding function for a standard mixture dis- 
tribution for i.i.d. data, as can be seen by comparison with (9.40) for the case of a 
Gaussian mixture. Here the quantities y(z,) are playing the role of the responsibil- 
ities. If the parameters @,, are independent for the different components, then this 
term decouples into a sum of terms one for each value of k, each of which can be 
maximized independently. We are then simply maximizing the weighted log likeli- 
hood function for the emission density p(x|@;,) with weights y(znx). Here we shall 
suppose that this maximization can be done efficiently. For instance, in the case of 
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Gaussian emission densities we have p(x|@,) = N (x| up, 5k), and maximization 
of the function Q(6, 0°") then gives 


mer- = (13.20) 


D, = 2 . (13.21) 


n=1 


For the case of discrete multinomial observed variables, the conditional distribution 
of the observations takes the form 


D K 
p(x|z) = lI i (13.22) 


and the corresponding M-step equations are given by 


N 
> 1enk) ni 
n=1 


N 
S Went) 


An analogous result holds for Bernoulli observed variables. 

The EM algorithm requires initial values for the parameters of the emission dis- 
tribution. One way to set these is first to treat the data initially as i.i.d. and fit the 
emission density by maximum likelihood, and then use the resulting values to ini- 
tialize the parameters for EM. 


13.2.2 The forward-backward algorithm 


Next we seek an efficient procedure for evaluating the quantities 7(zn,) and 
E(Zn—1,j, nk), corresponding to the E step of the EM algorithm. The graph for the 
hidden Markov model, shown in Figure 13.5, is a tree, and so we know that the 
posterior distribution of the latent variables can be obtained efficiently using a two- 
stage message passing algorithm. In the particular context of the hidden Markov 
model, this is known as the forward-backward algorithm (Rabiner, 1989), or the 
Baum-Welch algorithm (Baum, 1972). There are in fact several variants of the basic 
algorithm, all of which lead to the exact marginals, according to the precise form of 


Mik = (13.23) 
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the messages that are propagated along the chain (Jordan, 2007). We shall focus on 
the most widely used of these, known as the alpha-beta algorithm. 

As well as being of great practical importance in its own right, the forward- 
backward algorithm provides us with a nice illustration of many of the concepts 
introduced in earlier chapters. We shall therefore begin in this section with a ‘con- 
ventional’ derivation of the forward-backward equations, making use of the sum 
and product rules of probability, and exploiting conditional independence properties 
which we shall obtain from the corresponding graphical model using d-separation. 
Then in Section 13.2.3, we shall see how the forward-backward algorithm can be 
obtained very simply as a specific example of the sum-product algorithm introduced 
in Section 8.4.4. 

It is worth emphasizing that evaluation of the posterior distributions of the latent 
variables is independent of the form of the emission density p(x|z) or indeed of 
whether the observed variables are continuous or discrete. All we require is the 
values of the quantities p(x,,|z,,) for each value of Zn for every n. Also, in this 
section and the next we shall omit the explicit dependence on the model parameters 
6°" because these fixed throughout. 

We therefore begin by writing down the following conditional independence 
properties (Jordan, 2007) 


P(X|Zn) = plX1,---;XnlZn) 
Mj ids.n2<y XN|Zn) (13.24) 
Ol Mig iz Kiet | Mina) = Oi cess ya (A) (13.25) 
P(X: 205 Xn-1|Zn-1;Zn) = Pyne Maal Spe) (13.26) 
P(Xn41;-- -3 Ri Aas Baia) = PlXn+1;:--;XN|Zn+1) (13.27) 
POR ayers XN|Zn+1;Xn41) = PlXn42;---,XN|Zn+41) (13.28) 
pP(X|Zn-1;Zn) = p(Xi,...,Xn—-1|Zn-1) 
P(Xn|Zn)P(Xn+1, ---;XN|Zn) (13.29) 
P(Xn4ilX,Zn41) = p(Xn4i/Zn41) (13.30) 
plzn+ilzn, X) = p(zny+i|zn) (13.31) 
where X = {x1,..., Xy }. These relations are most easily proved using d-separation. 
For instance in the first of these results, we note that every path from any one of the 
nodes x1,...,Xn—1 to the node x, passes through the node z,,, which is observed. 


Because all such paths are head-to-tail, it follows that the conditional independence 
property must hold. The reader should take a few moments to verify each of these 
properties in turn, as an exercise in the application of d-separation. These relations 
can also be proved directly, though with significantly greater effort, from the joint 
distribution for the hidden Markov model using the sum and product rules of proba- 
bility. 

Let us begin by evaluating y(z,;). Recall that for a discrete multinomial ran- 
dom variable the expected value of one of its components is just the probability of 
that component having the value 1. Thus we are interested in finding the posterior 
distribution p(Zn|X1,...,X~) Of Zn given the observed data set x1,..., xm. This 
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represents a vector of length K whose entries correspond to the expected values of 
Znk. Using Bayes’ theorem, we have 


P(X|Zn)p(Zn) 
p(X) 


Note that the denominator p(X) is implicitly conditioned on the parameters 9°" 
of the HMM and hence represents the likelihood function. Using the conditional 
independence property (13.24), together with the product rule of probability, we 
obtain 


(Zn) = p(Zn|X) = (13.32) 


p(xı, see) Xn; Zn)P(Xn+1, tee ,Xn|Zn) A(Zn) B(Zn) 
J= = 13.33 
Wen) pX) p(X) iad 
where we have defined 
Q(Zn) = p(ki,-.-,Xn;Zn) (13.34) 
Blan) = P(¥n4i,---;Xn|Zn)- (13.35) 


The quantity a(z,,) represents the joint probability of observing all of the given 
data up to time n and the value of z,,, whereas (3(z,,) represents the conditional 
probability of all future data from time n + 1 up to N given the value of z,,. Again, 
a(Zn) and B(Zn) each represent set of K numbers, one for each of the possible 
settings of the 1-of- coded binary vector Zn. We shall use the notation a(zn%) to 
denote the value of a(z,,) when z,,~ = 1, with an analogous interpretation of (zng). 

We now derive recursion relations that allow a(z,,) and 3(z,,) to be evaluated 
efficiently. Again, we shall make use of conditional independence properties, in 
particular (13.25) and (13.26), together with the sum and product rules, allowing us 
to express a(zZ,,) in terms of a(Zn—1 ) as follows 


Zn). = p(Xi,..-;Xn,Zn) 
= p(xı, eee ,Xn|Zn)p(Zn) 
= P(Xn Zn)p(xı, boa pM ley Olas) 
= D(Xn Zn)P(X1, see, Xn—1; Zn) 
> P(Xn Zn) SG niinn] 
Zn—1 


a P(Xn Zn) `> p(xı, -3 Xn—1,; Zn|Zn—1)P(Zn—1) 


= p(%nlzn) Y p(xi,...;%n—1l2n—1)P(ZnlZn—1)P(Zn—1) 


= P(Xn Zn) 2 p(xı, -3 Xn—1,; Zn—1)P(Zn|Zn—1) 


Making use of the definition (13.34) for a(z,,), we then obtain 
a(Zn) = P(Xn|Zn) r9 a(Zn—1)P(Zn|Zn-1). (13.36) 


Zn—1 


Figure 13.12 
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Illustration of the forward recursion (13.36) for = a(zn_1,1) a(Zn,1) 
evaluation of the a variables. In this fragment A1 l 
of the lattice, we see that the quantity a(znı) k= 1 
is obtained by taking the elements a(zn—1,;) of 
a(Zn—1) at step n— 1 and summing them up with 
weights given by Aji, corresponding to the val- 

ues of p(Zn|zn—1), and then multiplying by the = @(2n—1,2) 
data contribution p(Xn|zn1). bed 


It is worth taking a moment to study this recursion relation in some detail. Note 
that there are K terms in the summation, and the right-hand side has to be evaluated 
for each of the K values of z,, so each step of the a recursion has computational 
cost that scaled like O(K?). The forward recursion equation for a(z,,) is illustrated 
using a lattice diagram in Figure 13.12. 

In order to start this recursion, we need an initial condition that is given by 


K 
a(zı) = p(x1, z1) = p(zı)p(xı|z1) = | | (meelo) (13.37) 


k=1 


which tells us that a(z1,), for k = 1,..., K, takes the value 7,p(xi|@;,,). Starting 
at the first node of the chain, we can then work along the chain and evaluate a(Zn) 
for every latent node. Because each step of the recursion involves multiplying by a 
K x K matrix, the overall cost of evaluating these quantities for the whole chain is 
of O(K?7N). 

We can similarly find a recursion relation for the quantities G(z,) by making 
use of the conditional independence properties (13.27) and (13.28) giving 


Baa) = P(Xn+1;---,XN|Zn) 
= Y P sAN Zaila) 
Zn+1 


= 2 P(Xn+1,- - -3 XN |Zn, Zn+1)P(Zn+1|Zn) 


= `> P(Xn41; see XN Zn+1)P(Zn41|Zn) 


= Y plXn tay KN Zn+1)P(Xn+1|Zn+1)P(Zn+1|Zn)- 
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Figure 13.13 Illustration of the backward recursion B(Zn1) B(2n41.1) 
(13.38) for evaluation of the 8 variables. In . Av 


this fragment of the lattice, we see that the k = 1 
quantity G(zni) is obtained by taking the 
components 6(2n+41,%) Of G(zn41) at step 

n + 1 and summing them up with weights 

given by the products of Aix, correspond- 

ing to the values of p(Zn+1|Zn) and the cor- 

responding values of the emission density k=2 g.. 
P(Xn|Zn41,k)- 


N D(Xn |2n41,1) 
B(Zn-+41,2) 


Ny 
B(Zn+1,3) 


Xn|Zn41,2) 


-0 
n 


n+ y 


Xn|2n41,3) 


Making use of the definition (13.35) for G(z,,), we then obtain 


Bln) = X B(n41)P(%n+1/2n41)P(2n+1/2n)- (13.38) 


Zn+1 


Note that in this case we have a backward message passing algorithm that evaluates 
B(Zn) in terms of 8(Zn+1). At each step, we absorb the effect of observation x, +1 
through the emission probability p(X,41|Zn41), multiply by the transition matrix 
P(Zn+1|Zn), and then marginalize out z,,,1. This is illustrated in Figure 13.13. 

Again we need a starting condition for the recursion, namely a value for G(zy). 
This can be obtained by setting n = N in (13.33) and replacing a(zy) with its 
definition (13.34) to give 


P(X, Zn) B(zn) 
p(X) 
which we see will be correct provided we take 3(z),) = 1 for all settings of zy. 


In the M step equations, the quantity p(X) will cancel out, as can be seen, for 
instance, in the M-step equation for u, given by (13.20), which takes the form 


p(zn|X) = (13.39) 


> Y(Zrik)Xn > Ot Bink) Bl 2nk) Xr 
a= == . (13.40) 
S Wem) J Alen) blnk) 
n=1 n=1 


However, the quantity p(X) represents the likelihood function whose value we typ- 
ically wish to monitor during the EM optimization, and so it is useful to be able to 
evaluate it. If we sum both sides of (13.33) over Zn, and use the fact that the left-hand 
side is a normalized distribution, we obtain 


p(X) = So a(n) B(2n): (13.41) 


Zn 
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Thus we can evaluate the likelihood function by computing this sum, for any conve- 
nient choice of n. For instance, if we only want to evaluate the likelihood function, 
then we can do this by running the a recursion from the start to the end of the chain, 
and then use this result for n = N, making use of the fact that 8(z y ) is a vector of 
1s. In this case no ( recursion is required, and we simply have 


p(X) = Yaz). (13.42) 


ZN 


Let us take a moment to interpret this result for p(X). Recall that to compute the 
likelihood we should take the joint distribution p(X, Z) and sum over all possible 
values of Z. Each such value represents a particular choice of hidden state for every 
time step, in other words every term in the summation is a path through the lattice 
diagram, and recall that there are exponentially many such paths. By expressing 
the likelihood function in the form (13.42), we have reduced the computational cost 
from being exponential in the length of the chain to being linear by swapping the 
order of the summation and multiplications, so that at each time step n we sum 
the contributions from all paths passing through each of the states z,;, to give the 
intermediate quantities a(z,,). 

Next we consider the evaluation of the quantities €(Zn—1, Zn), which correspond 
to the values of the conditional probabilities p(Zn—1, Zn|X) for each of the K x K 
settings for (Z,_1,Z,). Using the definition of €(z,,_1,Z,,), and applying Bayes’ 
theorem, we have 


E(Zn-i; Zn) = D(Zn—1, Zn |X) 
p(X|Zn—1, Zn)p(Zn-1, Zn) 


p(X) 
O Wn ailn 1)PEnlZn) Pn: XH ln) PCC na )PCOna) 
p(X) 
Ol Zi yO Ky Sy, OS Bi BS.) 
a (13.43) 


where we have made use of the conditional independence property (13.29) together 
with the definitions of a(z,,) and 3(z,,) given by (13.34) and (13.35). Thus we can 
calculate the €(z,,_1,Z,,) directly by using the results of the a and ( recursions. 
Let us summarize the steps required to train a hidden Markov model using 
the EM algorithm. We first make an initial selection of the parameters 0°’ where 
0 = (m, A, p). The A and m parameters are often initialized either uniformly or 
randomly from a uniform distribution (respecting their non-negativity and summa- 
tion constraints). Initialization of the parameters @ will depend on the form of the 
distribution. For instance in the case of Gaussians, the parameters u, might be ini- 
tialized by applying the k-means algorithm to the data, and Xx might be initialized 
to the covariance matrix of the corresponding K means cluster. Then we run both 
the forward a recursion and the backward ( recursion and use the results to evaluate 
(Zn) and €(Z,_1,Z,). At this stage, we can also evaluate the likelihood function. 
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This completes the E step, and we use the results to find a revised set of parameters 
6"” using the M-step equations from Section 13.2.1. We then continue to alternate 
between E and M steps until some convergence criterion is satisfied, for instance 
when the change in the likelihood function is below some threshold. 

Note that in these recursion relations the observations enter through conditional 
distributions of the form p(Xn|Zn). The recursions are therefore independent of 
the type or dimensionality of the observed variables or the form of this conditional 
distribution, so long as its value can be computed for each of the K possible states 
of Zn. Since the observed variables {x,,} are fixed, the quantities p(x,,|Z,,) can be 
pre-computed as functions of z,, at the start of the EM algorithm, and remain fixed 
throughout. 

We have seen in earlier chapters that the maximum likelihood approach is most 
effective when the number of data points is large in relation to the number of parame- 
ters. Here we note that a hidden Markov model can be trained effectively, using max- 
imum likelihood, provided the training sequence is sufficiently long. Alternatively, 
we can make use of multiple shorter sequences, which requires a straightforward 
modification of the hidden Markov model EM algorithm. In the case of left-to-right 
models, this is particularly important because, in a given observation sequence, a 
given state transition corresponding to a nondiagonal element of A will seen at most 
once. 

Another quantity of interest is the predictive distribution, in which the observed 
data is X = {x1,...,x.} and we wish to predict x, 41, which would be important 
for real-time applications such as financial forecasting. Again we make use of the 
sum and product rules together with the conditional independence properties (13.29) 
and (13.31) giving 


p(xwsil&) = >> pan znalX) 
ZN+1 
= » P(Xn41|2n 41) P(Zn41|X) 
ZN+1 
= 5 p(xn41|ZN41) S "plana, Zn |X) 
ZN+1 ZN 
= X plannlzny) X plansilzw)p(n |X) 
ZN+1 ZN 
ZN, X 
= J plxvsilznss) Y penlan) ERE 
ZN+1 ZN 
1 
= 500 XO p(xwsilzws1) X plawsilaw)a(zn) (13.44) 
ZN+1 ZN 


which can be evaluated by first running a forward a recursion and then computing 
the final summations over z y and Z y+1. The result of the first summation over Z y 
can be stored and used once the value of xXy+1 is observed in order to run the a 
recursion forward to the next step in order to predict the subsequent value xy+2. 
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Figure 13.14 A fragment of the fac- Z1 Zn—1 Wn Zn 
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Note that in (13.44), the influence of all data from x, to xy is summarized in the K 
values of a(zy ). Thus the predictive distribution can be carried forward indefinitely 
using a fixed amount of storage, as may be required for real-time applications. 

Here we have discussed the estimation of the parameters of an HMM using max- 
imum likelihood. This framework is easily extended to regularized maximum likeli- 
hood by introducing priors over the model parameters m, A and @ whose values are 
then estimated by maximizing their posterior probability. This can again be done us- 
ing the EM algorithm in which the E step is the same as discussed above, and the M 
step involves adding the log of the prior distribution p(@) to the function Q(6, 0°") 
before maximization and represents a straightforward application of the techniques 
developed at various points in this book. Furthermore, we can use variational meth- 
ods to give a fully Bayesian treatment of the HMM in which we marginalize over the 
parameter distributions (MacKay, 1997). As with maximum likelihood, this leads to 
a two-pass forward-backward recursion to compute posterior probabilities. 


13.2.3 The sum-product algorithm for the HMM 


The directed graph that represents the hidden Markov model, shown in Fig- 
ure 13.5, is a tree and so we can solve the problem of finding local marginals for the 
hidden variables using the sum-product algorithm. Not surprisingly, this turns out to 
be equivalent to the forward-backward algorithm considered in the previous section, 
and so the sum-product algorithm therefore provides us with a simple way to derive 
the alpha-beta recursion formulae. 

We begin by transforming the directed graph of Figure 13.5 into a factor graph, 
of which a representative fragment is shown in Figure 13.14. This form of the fac- 
tor graph shows all variables, both latent and observed, explicitly. However, for 
the purpose of solving the inference problem, we shall always be conditioning on 
the variables x,,...,X,y, and so we can simplify the factor graph by absorbing the 
emission probabilities into the transition probability factors. This leads to the sim- 
plified factor graph representation in Figure 13.15, in which the factors are given 
by 


h(zı) = p(zı)p(xı|zı) (13.45) 
Ín(Zn-1, Zn) P(Zn|Zn—1)P(Xn|Zn)- (13.46) 
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model. 
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To derive the alpha-beta algorithm, we denote the final hidden variable zy as 
the root node, and first pass messages from the leaf node h to the root. From the 
general results (8.66) and (8.69) for message propagation, we see that the messages 
which are propagated in the hidden Markov model take the form 


Mag 1—+fq (Zn-1) = Lfn—1—mn—1 (Zn—-1) (13.47) 
aa) D Jeta Perea) A) 


These equations represent the propagation of messages forward along the chain and 
are equivalent to the alpha recursions derived in the previous section, as we shall 
now show. Note that because the variable nodes z,, have only two neighbours, they 
perform no computation. 

We can eliminate Hz„_ı— fn (Zn—1) from (13.48) using (13.47) to give a recur- 
sion for the f — z messages of the form 


Lf, an (Zn) = bD Ín(Zn-1; Zn) fn —1—en—1 (Zn—-1)- (13.49) 


Zn—1 


If we now recall the definition (13.46), and if we define 


Gn) = heen) (13.50) 


then we obtain the alpha recursion given by (13.36). We also need to verify that 
the quantities a(z,,) are themselves equivalent to those defined previously. This 
is easily done by using the initial condition (8.71) and noting that a(z,) is given 
by h(zı) = p(z1)p(x1|Z1) which is identical to (13.37). Because the initial a is 
the same, and because they are iteratively computed using the same equation, all 
subsequent a quantities must be the same. 

Next we consider the messages that are propagated from the root node back to 
the leaf node. These take the form 


Ufny1—>fn (Zn) = > FnsilZn,2n41) frre fags (Zn41) (13.51) 


Zn+1 


where, as before, we have eliminated the messages of the type z — f since the 
variable nodes perform no computation. Using the definition (13.46) to substitute 
for fn+1(Zn, Zn+1), and defining 


BlZn) = Hfnpi—>zn (Zn) (13.52) 
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we obtain the beta recursion given by (13.38). Again, we can verify that the beta 
variables themselves are equivalent by noting that (8.70) implies that the initial mes- 
sage send by the root variable node is z,,fy (ZN) = 1, which is identical to the 
initialization of 3(zy) given in Section 13.2.2. 

The sum-product algorithm also specifies how to evaluate the marginals once all 
the messages have been evaluated. In particular, the result (8.63) shows that the local 
marginal at the node z,, is given by the product of the incoming messages. Because 


we have conditioned on the variables X = {x1,...,x.}, we are computing the 
joint distribution 
PlZn, X) = [fy vty (Zn) Mf 1an (Zn) = O(Zn)/B(Zn): (13.53) 


Dividing both sides by p(X), we then obtain 


P(Zn,X) — a(Zn) (Zn) 


Wen) =) CX) ame 


in agreement with (13.33). The result (13.43) can similarly be derived from (8.72). 


13.2.4 Scaling factors 


There is an important issue that must be addressed before we can make use of the 
forward backward algorithm in practice. From the recursion relation (13.36), we note 
that at each step the new value a(z,,) is obtained from the previous value a(zn—1) 
by multiplying by quantities p(z,,|Z,_1) and p(x,,|Z,,). Because these probabilities 
are often significantly less than unity, as we work our way forward along the chain, 
the values of a(z,,) can go to zero exponentially quickly. For moderate lengths of 
chain (say 100 or so), the calculation of the a(z,,) will soon exceed the dynamic 
range of the computer, even if double precision floating point is used. 

In the case of i.i.d. data, we implicitly circumvented this problem with the eval- 
uation of likelihood functions by taking logarithms. Unfortunately, this will not help 
here because we are forming sums of products of small numbers (we are in fact im- 
plicitly summing over all possible paths through the lattice diagram of Figure 13.7). 
We therefore work with re-scaled versions of a(z,) and (Zn) whose values remain 
of order unity. As we shall see, the corresponding scaling factors cancel out when 
we use these re-scaled quantities in the EM algorithm. 

In (13.34), we defined a(Zn) = p(X1,.--,Xn,Zn) representing the joint distri- 
bution of all the observations up to x,, and the latent variable z,,. Now we define a 
normalized version of a given by 

Q(Zn) = PlZn|X1,. Xn) = n) (13.55) 
” ” i _ p(X1,---,Xn) 
which we expect to be well behaved numerically because it is a probability distribu- 
tion over K variables for any value of n. In order to relate the scaled and original al- 
pha variables, we introduce scaling factors defined by conditional distributions over 
the observed variables 
Ce = Wy Riga sy pan): (13.56) 
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From the product rule, we then have 


Pia) = [on (13.57) 
m=] 
and so 


Q(Zn) = P(Zn|X1,---,Xn)P(X1,---;Xn) = (Ù on) A(Zn). (13.58) 


m=1 
We can then turn the recursion equation (13.36) for a into one for @ given by 
Cna(Zn) = p(Xn|Zn) 5 Q(Zn—1)P(Zn|Zn—1)- (13.59) 
Zn-1 


Note that at each stage of the forward message passing phase, used to evaluate a(z,,), 
we have to evaluate and store cn, which is easily done because it is the coefficient 
that normalizes the right-hand side of (13.59) to give A(z»). 


We can similarly define re-scaled variables B(Zn) using 


N 
B(Zn) = ( II 3 Blzn) (13.60) 


m=n+1 


which will again remain within machine precision because, from (13.35), the quan- 
tities G(z,,) are simply the ratio of two conditional probabilities 


AE e (13.61) 
P(Xn+1, nee »Xn|X1, aaa Xn) 


The recursion result (13.38) for 8 then gives the following recursion for the re-scaled 


variables ue pa 
Cn+1b(Zn) = 5 B(Zn H 1)P(Xn } 1|Zn ' 1)P(Zn 1|Zn). (13.62) 


Zn+1 


In applying this recursion relation, we make use of the scaling factors cn that were 
previously computed in the a phase. 
From (13.57), we see that the likelihood function can be found using 


p(X) = |f en. (13.63) 


Similarly, using (13.33) and (13.43), together with (13.63), we see that the required 
Exercise 13.15 marginals are given by 


Y(Zn) = G(Zn)B(zn) (13.64) 
€(Zn—1,Zn) = Cn@(Zn—1)p(Xn|2n)p(Zn|Z—1) B(Zn). (13.65) 
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Finally, we note that there is an alternative formulation of the forward-backward 
algorithm (Jordan, 2007) in which the backward pass is defined by a recursion based 
the quantities y(z,) = Q(Zp)G(Z,) instead of using 3(z,). This a~y recursion 
requires that the forward pass be completed first so that all the quantities a(z,,) 
are available for the backward pass, whereas the forward and backward passes of 
the a—( algorithm can be done independently. Although these two algorithms have 
comparable computational cost, the a—G version is the most commonly encountered 
one in the case of hidden Markov models, whereas for linear dynamical systems a 
recursion analogous to the a—y form is more usual. 


13.2.5 The Viterbi algorithm 


In many applications of hidden Markov models, the latent variables have some 
meaningful interpretation, and so it is often of interest to find the most probable 
sequence of hidden states for a given observation sequence. For instance in speech 
recognition, we might wish to find the most probable phoneme sequence for a given 
series of acoustic observations. Because the graph for the hidden Markov model is 
a directed tree, this problem can be solved exactly using the max-sum algorithm. 
We recall from our discussion in Section 8.4.5 that the problem of finding the most 
probable sequence of latent states is not the same as that of finding the set of states 
that are individually the most probable. The latter problem can be solved by first 
running the forward-backward (sum-product) algorithm to find the latent variable 
marginals 7(z,,) and then maximizing each of these individually (Duda et al., 2001). 
However, the set of such states will not, in general, correspond to the most probable 
sequence of states. In fact, this set of states might even represent a sequence having 
zero probability, if it so happens that two successive states, which in isolation are 
individually the most probable, are such that the transition matrix element connecting 
them is zero. 

In practice, we are usually interested in finding the most probable sequence of 
states, and this can be solved efficiently using the max-sum algorithm, which in the 
context of hidden Markov models is known as the Viterbi algorithm (Viterbi, 1967). 
Note that the max-sum algorithm works with log probabilities and so there is no 
need to use re-scaled variables as was done with the forward-backward algorithm. 
Figure 13.16 shows a fragment of the hidden Markov model expanded as lattice 
diagram. As we have already noted, the number of possible paths through the lattice 
grows exponentially with the length of the chain. The Viterbi algorithm searches this 
space of paths efficiently to find the most probable path with a computational cost 
that grows only linearly with the length of the chain. 

As with the sum-product algorithm, we first represent the hidden Markov model 
as a factor graph, as shown in Figure 13.15. Again, we treat the variable node z y 
as the root, and pass messages to the root starting with the leaf nodes. Using the 
results (8.93) and (8.94), we see that the messages passed in the max-sum algorithm 
are given by 


Pan—fnyi(Zn) = Ufn—an (Zn) (13.66) 
hfn tings (Zn41) = max {In fn4i(Zn, Zn+1) F Man fn+1 (Zn) } . (13.67) 


Zn 
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Figure 13.16 A fragment of the HMM lattice 
showing two possible paths. The Viterbi algorithm 


any given path, the corresponding probability is 

given by the product of the elements of the tran- 

sition matrix A;,, Corresponding to the probabil- 

ities p(Zn+ı|Zn) for each segment of the path, 

along with the emission densities p(x,,|k) asso- k = 2 
ciated with each node on the path. 
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——————_—_= > 
efficiently determines the most probable path from k = 1 go g g eee 
amongst the exponentially many possibilities. For 

n-1 n 


If we eliminate Hz, — f,,,, (Zn) between these two equations, and make use of (13.46), 
we obtain a recursion for the f — z messages of the form 


wW(Zn41) = Inp(Xn41|Zn41) + max {In p(x+1|Zn) + w(Zn)} (13.68) 


where we have introduced the notation w(Zn) = Hf, +2, (Zn). 
From (8.95) and (8.96), these messages are initialized using 


w(Z1) = ln p(z1) + In p(x,|z1). (13.69) 


where we have used (13.45). Note that to keep the notation uncluttered, we omit 
the dependence on the model parameters @ that are held fixed when finding the most 
probable sequence. 

The Viterbi algorithm can also be derived directly from the definition (13.6) of 
the joint distribution by taking the logarithm and then exchanging maximizations 
and summations. It is easily seen that the quantities w(z,,) have the probabilistic 
interpretation 

wW(Zn)= max p(Xi,...,Xn,Z1,---,Zn)- (13.70) 


Once we have completed the final maximization over zy, we will obtain the 
value of the joint distribution p(X, Z) corresponding to the most probable path. We 
also wish to find the sequence of latent variable values that corresponds to this path. 
To do this, we simply make use of the back-tracking procedure discussed in Sec- 
tion 8.4.5. Specifically, we note that the maximization over z,, must be performed 
for each of the K possible values of z,,,,. Suppose we keep a record of the values 
of Z, that correspond to the maxima for each value of the K values of z,,,1. Let us 
denote this function by %(kn) where k € {1,..., K}. Once we have passed mes- 
sages to the end of the chain and found the most probable state of zy, we can then 
use this function to backtrack along the chain by applying it recursively 


hme = oh (KM), (13.71) 
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Intuitively, we can understand the Viterbi algorithm as follows. Naively, we 
could consider explicitly all of the exponentially many paths through the lattice, 
evaluate the probability for each, and then select the path having the highest proba- 
bility. However, we notice that we can make a dramatic saving in computational cost 
as follows. Suppose that for each path we evaluate its probability by summing up 
products of transition and emission probabilities as we work our way forward along 
each path through the lattice. Consider a particular time step n and a particular state 
k at that time step. There will be many possible paths converging on the correspond- 
ing node in the lattice diagram. However, we need only retain that particular path 
that so far has the highest probability. Because there are K states at time step n, we 
need to keep track of K such paths. At time step n + 1, there will be K? possible 
paths to consider, comprising K possible paths leading out of each of the K current 
states, but again we need only retain K of these corresponding to the best path for 
each state at time n+ 1. When we reach the final time step N we will discover which 
state corresponds to the overall most probable path. Because there is a unique path 
coming into that state we can trace the path back to step N — 1 to see what state it 
occupied at that time, and so on back through the lattice to the state n = 1. 


13.2.6 Extensions of the hidden Markov model 


The basic hidden Markov model, along with the standard training algorithm 
based on maximum likelihood, has been extended in numerous ways to meet the 
requirements of particular applications. Here we discuss a few of the more important 
examples. 

We see from the digits example in Figure 13.11 that hidden Markov models can 
be quite poor generative models for the data, because many of the synthetic digits 
look quite unrepresentative of the training data. If the goal is sequence classifica- 
tion, there can be significant benefit in determining the parameters of hidden Markov 
models using discriminative rather than maximum likelihood techniques. Suppose 
we have a training set of R observation sequences X,, where r = 1,..., R, each of 
which is labelled according to its class m, where m = 1,...,M. For each class, we 
have a separate hidden Markov model with its own parameters 0m, and we treat the 
problem of determining the parameter values as a standard classification problem in 
which we optimize the cross-entropy 


R 
S"Inp(m,|X,). (13.72) 


Using Bayes’ theorem this can be expressed in terms of the sequence probabilities 
associated with the hidden Markov models 


R 
Son E pX, (Orpi) | (13.73) 


rat (Ek (Kr 16) v(t) 


where p(m) is the prior probability of class m. Optimization of this cost function 
is more complex than for maximum likelihood (Kapadia, 1998), and in particular 


632 


Figure 13.17 Section of an autoregressive hidden 


13. SEQUENTIAL DATA 


Markov model, in which the distribution 
of the observation x, depends on a 
subset of the previous observations as 
well as on the hidden state z». In this 
example, the distribution of x, depends 
on the two previous observations x,,_1 
and xn-2. 


requires that every training sequence be evaluated under each of the models in or- 
der to compute the denominator in (13.73). Hidden Markov models, coupled with 
discriminative training methods, are widely used in speech recognition (Kapadia, 
1998). 

A significant weakness of the hidden Markov model is the way in which it rep- 
resents the distribution of times for which the system remains in a given state. To see 
the problem, note that the probability that a sequence sampled from a given hidden 
Markov model will spend precisely T steps in state k and then make a transition to a 
different state is given by 


p(T) = (Agr)? (1 — Akk) X exp (-T ln Akk) (13.74) 


and so is an exponentially decaying function of T. For many applications, this will 
be a very unrealistic model of state duration. The problem can be resolved by mod- 
elling state duration directly in which the diagonal coefficients A% are all set to zero, 
and each state k is explicitly associated with a probability distribution p(T|k) of pos- 
sible duration times. From a generative point of view, when a state k is entered, a 
value T representing the number of time steps that the system will remain in state k 
is then drawn from p(T|k). The model then emits T values of the observed variable 
X, which are generally assumed to be independent so that the corresponding emis- 
sion density is simply Ha p(x:|k). This approach requires some straightforward 
modifications to the EM optimization procedure (Rabiner, 1989). 

Another limitation of the standard HMM is that it is poor at capturing long- 
range correlations between the observed variables (i.e., between variables that are 
separated by many time steps) because these must be mediated via the first-order 
Markov chain of hidden states. Longer-range effects could in principle be included 
by adding extra links to the graphical model of Figure 13.5. One way to address this 
is to generalize the HMM to give the autoregressive hidden Markov model (Ephraim 
et al., 1989), an example of which is shown in Figure 13.17. For discrete observa- 
tions, this corresponds to expanded tables of conditional probabilities for the emis- 
sion distributions. In the case of a Gaussian emission density, we can use the linear- 
Gaussian framework in which the conditional distribution for xn given the values 
of the previous observations, and the value of z,,, is a Gaussian whose mean is a 
linear combination of the values of the conditioning variables. Clearly the number 
of additional links in the graph must be limited to avoid an excessive the number of 
free parameters. In the example shown in Figure 13.17, each observation depends on 
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Figure 13.18 Example of an_ input-output hidden Un—-1 Un Un+1 
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Markov model. In this case, both the 
emission probabilities and the transition 
probabilities depend on the values of a 
sequence of observations u1,..., uv. 


the two preceding observed variables as well as on the hidden state. Although this 
graph looks messy, we can again appeal to d-separation to see that in fact it still has 
a simple probabilistic structure. In particular, if we imagine conditioning on z,, we 
see that, as with the standard HMM, the values of z,_; and Zn; are independent, 
corresponding to the conditional independence property (13.5). This is easily veri- 
fied by noting that every path from node z,,_; to node z,,4; passes through at least 
one observed node that is head-to-tail with respect to that path. As a consequence, 
we can again use a forward-backward recursion in the E step of the EM algorithm to 
determine the posterior distributions of the latent variables in a computational time 
that is linear in the length of the chain. Similarly, the M step involves only a minor 
modification of the standard M-step equations. In the case of Gaussian emission 
densities this involves estimating the parameters using the standard linear regression 
equations, discussed in Chapter 3. 

We have seen that the autoregressive HMM appears as a natural extension of the 
standard HMM when viewed as a graphical model. In fact the probabilistic graphical 
modelling viewpoint motivates a plethora of different graphical structures based on 
the HMM. Another example is the input-output hidden Markov model (Bengio and 
Frasconi, 1995), in which we have a sequence of observed variables u,,..., uy, in 
addition to the output variables x;,...,x,, whose values influence either the dis- 
tribution of latent variables or output variables, or both. An example is shown in 
Figure 13.18. This extends the HMM framework to the domain of supervised learn- 
ing for sequential data. It is again easy to show, through the use of the d-separation 
criterion, that the Markov property (13.5) for the chain of latent variables still holds. 
To verify this, simply note that there is only one path from node z,,_; to node z,,1 
and this is head-to-tail with respect to the observed node z,,. This conditional inde- 
pendence property again allows the formulation of a computationally efficient learn- 
ing algorithm. In particular, we can determine the parameters 0 of the model by 
maximizing the likelihood function L(@) = p(X|U, @) where U is a matrix whose 
rows are given by uf. As a consequence of the conditional independence property 
(13.5) this likelihood function can be maximized efficiently using an EM algorithm 
in which the E step involves forward and backward recursions. 

Another variant of the HMM worthy of mention is the factorial hidden Markov 
model (Ghahramani and Jordan, 1997), in which there are multiple independent 
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Figure 13.19 


Figure 13.20 


A factorial hidden Markov model com- 
prising two Markov chains of latent vari- 
ables. For continuous observed variables 
x, one possible choice of emission model 
is a linear-Gaussian density in which the 
mean of the Gaussian is a linear combi- 
nation of the states of the corresponding 
latent variables. 


Markov chains of latent variables, and the distribution of the observed variable at 
a given time step is conditional on the states of all of the corresponding latent vari- 
ables at that same time step. Figure 13.19 shows the corresponding graphical model. 
The motivation for considering factorial HMM can be seen by noting that in order to 
represent, say, 10 bits of information at a given time step, a standard HMM would 
need K = 2!° = 1024 latent states, whereas a factorial HMM could make use of 10 
binary latent chains. The primary disadvantage of factorial HMMs, however, lies in 
the additional complexity of training them. The M step for the factorial HMM model 
is straightforward. However, observation of the x variables introduces dependencies 
between the latent chains, leading to difficulties with the E step. This can be seen 
by noting that in Figure 13.19, the variables zi) and zi?) are connected by a path 
which is head-to-head at node x,, and hence they are not d-separated. The exact E 
step for this model does not correspond to running forward and backward recursions 
along the M Markov chains independently. This is confirmed by noting that the key 
conditional independence property (13.5) is not satisfied for the individual Markov 
chains in the factorial HMM model, as is shown using d-separation in Figure 13.20. 
Now suppose that there are M chains of hidden nodes and for simplicity suppose 
that all latent variables have the same number K of states. Then one approach would 
be to note that there are K™ combinations of latent variables at a given time step 


Example of a path, highlighted in green, 
which is head-to-head at the observed 
nodes x,-1 and xn+ı, and head-to-tail 


at the unobserved nodes 2°”), , z? and 


n—1’ 
z‘°),. Thus the path is not blocked and 


so the conditional independence property 
(13.5) does not hold for the individual la- 
tent chains of the factorial HMM model. 
As a consequence, there is no efficient 
exact E step for this model. 
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and so we can transform the model into an equivalent standard HMM having a single 
chain of latent variables each of which has K™ latent states. We can then run the 
standard forward-backward recursions in the E step. This has computational com- 
plexity O(N K?™) that is exponential in the number M of latent chains and so will 
be intractable for anything other than small values of M. One solution would be 
to use sampling methods (discussed in Chapter 11). As an elegant deterministic al- 
ternative, Ghahramani and Jordan (1997) exploited variational inference techniques 
to obtain a tractable algorithm for approximate inference. This can be done using 
a simple variational posterior distribution that is fully factorized with respect to the 
latent variables, or alternatively by using a more powerful approach in which the 
variational distribution is described by independent Markov chains corresponding to 
the chains of latent variables in the original model. In the latter case, the variational 
inference algorithms involves running independent forward and backward recursions 
along each chain, which is computationally efficient and yet is also able to capture 
correlations between variables within the same chain. 

Clearly, there are many possible probabilistic structures that can be constructed 
according to the needs of particular applications. Graphical models provide a general 
technique for motivating, describing, and analysing such structures, and variational 
methods provide a powerful framework for performing inference in those models for 
which exact solution is intractable. 


Linear Dynamical Systems 


In order to motivate the concept of linear dynamical systems, let us consider the 
following simple problem, which often arises in practical settings. Suppose we wish 
to measure the value of an unknown quantity z using a noisy sensor that returns a 
observation x representing the value of z plus zero-mean Gaussian noise. Given a 
single measurement, our best guess for z is to assume that z = x. However, we 
can improve our estimate for z by taking lots of measurements and averaging them, 
because the random noise terms will tend to cancel each other. Now let’s make the 
situation more complicated by assuming that we wish to measure a quantity z that 
is changing over time. We can take regular measurements of x so that at some point 
in time we have obtained x;,...,x, and we wish to find the corresponding values 
Z1,...,Xn. If we simply average the measurements, the error due to random noise 
will be reduced, but unfortunately we will just obtain a single averaged estimate, in 
which we have averaged over the changing value of z, thereby introducing a new 
source of error. 

Intuitively, we could imagine doing a bit better as follows. To estimate the value 
of zy, we take only the most recent few measurements, say xy—p,...,XN and just 
average these. If z is changing slowly, and the random noise level in the sensor is 
high, it would make sense to choose a relatively long window of observations to 
average. Conversely, if the signal is changing quickly, and the noise levels are small, 
we might be better just to use xy directly as our estimate of zy. Perhaps we could 
do even better if we take a weighted average, in which more recent measurements 
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make a greater contribution than less recent ones. 

Although this sort of intuitive argument seems plausible, it does not tell us how 
to form a weighted average, and any sort of hand-crafted weighing is hardly likely 
to be optimal. Fortunately, we can address problems such as this much more sys- 
tematically by defining a probabilistic model that captures the time evolution and 
measurement processes and then applying the inference and learning methods devel- 
oped in earlier chapters. Here we shall focus on a widely used model known as a 
linear dynamical system. 

As we have seen, the HMM corresponds to the state space model shown in 
Figure 13.5 in which the latent variables are discrete but with arbitrary emission 
probability distributions. This graph of course describes a much broader class of 
probability distributions, all of which factorize according to (13.6). We now consider 
extensions to other distributions for the latent variables. In particular, we consider 
continuous latent variables in which the summations of the sum-product algorithm 
become integrals. The general form of the inference algorithms will, however, be 
the same as for the hidden Markov model. It is interesting to note that, historically, 
hidden Markov models and linear dynamical systems were developed independently. 
Once they are both expressed as graphical models, however, the deep relationship 
between them immediately becomes apparent. 

One key requirement is that we retain an efficient algorithm for inference which 
is linear in the length of the chain. This requires that, for instance, when we take 
a quantity Q(Z,_1), representing the posterior probability of z,, given observations 
X1,.--,;Xn, and multiply by the transition probability p(z,|Zn—1) and the emission 
probability p(Xn|Zn) and then marginalize over z,_1, we obtain a distribution over 
Zn that is of the same functional form as that over @(Z,—1). That is to say, the 
distribution must not become more complex at each stage, but must only change in 
its parameter values. Not surprisingly, the only distributions that have this property 
of being closed under multiplication are those belonging to the exponential family. 

Here we consider the most important example from a practical perspective, 
which is the Gaussian. In particular, we consider a linear-Gaussian state space model 
so that the latent variables {z,,}, as well as the observed variables {x,,}, are multi- 
variate Gaussian distributions whose means are linear functions of the states of their 
parents in the graph. We have seen that a directed graph of linear-Gaussian units 
is equivalent to a joint Gaussian distribution over all of the variables. Furthermore, 
marginals such as @(Z,,) are also Gaussian, so that the functional form of the mes- 
sages is preserved and we will obtain an efficient inference algorithm. By contrast, 
suppose that the emission densities p(x,,|z,,) comprise a mixture of K Gaussians 
each of which has a mean that is linear in z,,. Then even if a(z,) is Gaussian, the 
quantity Q(z») will be a mixture of K Gaussians, @(z3) will be a mixture of K? 
Gaussians, and so on, and exact inference will not be of practical value. 

We have seen that the hidden Markov model can be viewed as an extension of 
the mixture models of Chapter 9 to allow for sequential correlations in the data. 
In a similar way, we can view the linear dynamical system as a generalization of the 
continuous latent variable models of Chapter 12 such as probabilistic PCA and factor 
analysis. Each pair of nodes {Zn, Xn } represents a linear-Gaussian latent variable 
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model for that particular observation. However, the latent variables {z,,} are no 
longer treated as independent but now form a Markov chain. 

Because the model is represented by a tree-structured directed graph, inference 
problems can be solved efficiently using the sum-product algorithm. The forward re- 
cursions, analogous to the a messages of the hidden Markov model, are known as the 
Kalman filter equations (Kalman, 1960; Zarchan and Musoff, 2005), and the back- 
ward recursions, analogous to the 3 messages, are known as the Kalman smoother 
equations, or the Rauch-Tung-Striebel (RTS) equations (Rauch et al., 1965). The 
Kalman filter is widely used in many real-time tracking applications. 

Because the linear dynamical system is a linear-Gaussian model, the joint distri- 
bution over all variables, as well as all marginals and conditionals, will be Gaussian. 
It follows that the sequence of individually most probable latent variable values is 
the same as the most probable latent sequence. There is thus no need to consider the 
analogue of the Viterbi algorithm for the linear dynamical system. 

Because the model has linear-Gaussian conditional distributions, we can write 
the transition and emission distributions in the general form 


PlZn|Zn-1) = N(@,|Azn—1,T) (13.75) 
P(Xn|Zn) = N(x,|Cz,;%). (13.76) 


The initial latent variable also has a Gaussian distribution which we write as 
p(z1) = N (z1ı| u0, Vo). (13.77) 


Note that in order to simplify the notation, we have omitted additive constant terms 
from the means of the Gaussians. In fact, it is straightforward to include them if 
desired. Traditionally, these distributions are more commonly expressed in an equiv- 
alent form in terms of noisy linear equations given by 


Zn = AZn-1+Wn (13.78) 
Xn = CZ + Vn (13.79) 
Z = MW+u (13.80) 


where the noise terms have the distributions 


w ~ N(wļ0,T) (13.81) 
v ~ N(v|0,5) (13.82) 
u ~ N(ul0, Vo). (13.83) 


The parameters of the model, denoted by 80 = {A,T, C, ©, mo, Vo}, can be 
determined using maximum likelihood through the EM algorithm. In the E step, we 
need to solve the inference problem of determining the local posterior marginals for 
the latent variables, which can be solved efficiently using the sum-product algorithm, 
as we discuss in the next section. 
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13.3.1 Inference in LDS 


We now turn to the problem of finding the marginal distributions for the latent 
variables conditional on the observation sequence. For given parameter settings, we 
also wish to make predictions of the next latent state z,, and of the next observation 
Xn conditioned on the observed data x1,...,X,—1 for use in real-time applications. 
These inference problems can be solved efficiently using the sum-product algorithm, 
which in the context of the linear dynamical system gives rise to the Kalman filter 
and Kalman smoother equations. 

It is worth emphasizing that because the linear dynamical system is a linear- 
Gaussian model, the joint distribution over all latent and observed variables is simply 
a Gaussian, and so in principle we could solve inference problems by using the 
standard results derived in previous chapters for the marginals and conditionals of a 
multivariate Gaussian. The role of the sum-product algorithm is to provide a more 
efficient way to perform such computations. 

Linear dynamical systems have the identical factorization, given by (13.6), to 
hidden Markov models, and are again described by the factor graphs in Figures 13.14 
and 13.15. Inference algorithms therefore take precisely the same form except that 
summations over latent variables are replaced by integrations. We begin by consid- 
ering the forward equations in which we treat zy as the root node, and propagate 
messages from the leaf node h(z,) to the root. From (13.77), the initial message will 
be Gaussian, and because each of the factors is Gaussian, all subsequent messages 
will also be Gaussian. By convention, we shall propagate messages that are nor- 
malized marginal distributions corresponding to p(Z,|X1,...,Xn), which we denote 
by 

az) = N (Zal Bes Vn). (13.84) 


This is precisely analogous to the propagation of scaled variables a(z,,) given by 
(13.59) in the discrete case of the hidden Markov model, and so the recursion equa- 
tion now takes the form 


CrCl Zn) = p(Xy |Z) J Rens)plenlen-1) dZn—1.- (13.85) 


Substituting for the conditionals p(Zn|Zn—1) and p(Xn|Zn ), using (13.75) and (13.76), 
respectively, and making use of (13.84), we see that (13.85) becomes 


CnN (Zn| Hn, Vn) z N(Xn|CZn, x) 
[Nel Aen a DIN anilin- Va) dZn—1- (13.86) 


Here we are supposing that yz,,_, and V,,_; are known, and by evaluating the inte- 
gral in (13.86), we wish to determine values for jz,, and Vn. The integral is easily 
evaluated by making use of the result (2.115), from which it follows that 


[Nel Rena EN Colin ay Vint) dns 
= N (Zn|A n1, Pn-1) (13.87) 
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where we have defined 
Pn- = AV, AT 4T. (13.88) 


We can now combine this result with the first factor on the right-hand side of (13.86) 
by making use of (2.115) and (2.116) to give 


Un = Alnı +Krn(xn —CAp,_;) (13.89) 
V, = (1-K,C)P,-1 (13.90) 
Cn = Nix CAp,.,0P,4C’ + 5). (13.91) 


Here we have made use of the matrix inverse identities (C.5) and (C.7) and also 
defined the Kalman gain matrix 


Kyo PC (Che 4s). (13.92) 


Thus, given the values of ,,_, and V,,_1, together with the new observation Xp, 
we can evaluate the Gaussian marginal for z,, having mean u, and covariance Vn, 
as well as the normalization coefficient cn. 

The initial conditions for these recursion equations are obtained from 


€1Q(Z1) = p(Z1)p(x1|Z1). (13.93) 


Because p(z,) is given by (13.77), and p(x;|z1) is given by (13.76), we can again 
make use of (2.115) to calculate cı and (2.116) to calculate u, and V; giving 


Hı = bo + Ki (x1 — Cpo) (13.94) 
Vi = (I-K,C)Vo (13.95) 
a = N(x|Cyuy,CVoC™ + 5) (13.96) 
where E 
Kı = VC? (CVoCT +). (13.97) 


Similarly, the likelihood function for the linear dynamical system is given by (13.63) 
in which the factors c,, are found using the Kalman filtering equations. 

We can interpret the steps involved in going from the posterior marginal over 
Zn—1 to the posterior marginal over Zn as follows. In (13.89), we can view the 
quantity Ay,,_, as the prediction of the mean over z,, obtained by simply taking the 
mean over Zn—ı and projecting it forward one step using the transition probability 
matrix A. This predicted mean would give a predicted observation for x,, given by 
CAz,,—1 obtained by applying the emission probability matrix C to the predicted 
hidden state mean. We can view the update equation (13.89) for the mean of the 
hidden variable distribution as taking the predicted mean Ay, and then adding 
a correction that is proportional to the error xn — CAz,_; between the predicted 
observation and the actual observation. The coefficient of this correction is given by 
the Kalman gain matrix. Thus we can view the Kalman filter as a process of making 
successive predictions and then correcting these predictions in the light of the new 
observations. This is illustrated graphically in Figure 13.21. 
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Zn—1 Zn 


Figure 13.21 The linear dynamical system can be viewed as a sequence of steps in which increasing un- 
certainty in the state variable due to diffusion is compensated by the arrival of new data. In the left-hand plot, 
the blue curve shows the distribution p(Zn-1|X1,...,Xn—-1), which incorporates all the data up to step n — 1. 
The diffusion arising from the nonzero variance of the transition probability p(Zn|Zn-1) gives the distribution 
P(Zn|X1,---,Xn—1), Shown in red in the centre plot. Note that this is broader and shifted relative to the blue curve 
(which is shown dashed in the centre plot for comparison). The next data observation x,, contributes through the 
emission density p(x,,|Zn), which is shown as a function of z» in green on the right-hand plot. Note that this is not 
a density with respect to zn and so is not normalized to one. Inclusion of this new data point leads to a revised 
distribution p(zn|x1,...,Xn) for the state density shown in blue. We see that observation of the data has shifted 
and narrowed the distribution compared to p(z,|x1,...,Xn—1) (which is shown in dashed in the right-hand plot 
for comparison). 


If we consider a situation in which the measurement noise is small compared 

to the rate at which the latent variable is evolving, then we find that the posterior 
Exercise 13.27 distribution for z,, depends only on the current measurement x,,, in accordance with 

the intuition from our simple example at the start of the section. Similarly, if the 

latent variable is evolving slowly relative to the observation noise level, we find that 

the posterior mean for Z, is obtained by averaging all of the measurements obtained 
Exercise 13.28 up to that time. 

One of the most important applications of the Kalman filter is to tracking, and 
this is illustrated using a simple example of an object moving in two dimensions in 
Figure 13.22. 

So far, we have solved the inference problem of finding the posterior marginal 
for a node z,, given observations from x; up to xn. Next we turn to the problem of 
finding the marginal for a node z,, given all observations x; to xy. For temporal 
data, this corresponds to the inclusion of future as well as past observations. Al- 
though this cannot be used for real-time prediction, it plays a key role in learning the 
parameters of the model. By analogy with the hidden Markov model, this problem 
can be solved by propagating messages from node xy back to node x, and com- 
bining this information with that obtained during the forward message passing stage 
used to compute the a(Z,,). 

In the LDS literature, it is usual to formulate this backward recursion in terms 
of (Zn) = Q(Zp)G(Z») rather than in terms of 3(z,,). Because 7(z,,) must also be 
Gaussian, we write it in the form 


n 


(tin) = (Zn) 6 (an) = N (Zn|fin, Vn). (13.98) 


To derive the required recursion, we start from the backward recursion (13.62) for 


Figure 13.22 An illustration of a linear dy- 
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namical system being used to 
track a moving object. The blue 
points indicate the true positions 
of the object in a two-dimensional 
space at successive time steps, 
the green points denote noisy 
measurements of the positions, 
and the red crosses indicate the 
means of the inferred posterior 
distributions of the positions ob- 
tained by running the Kalman fil- 
tering equations. The covari- 
ances of the inferred positions 
are indicated by the red ellipses, 
which correspond to contours 
having one standard deviation. 


B(Zn), which, for continuous latent variables, can be written in the form 


Cn+13 (Zn) = | Blenss)Pl nessa) PCC tin) dens (13.99) 


We now multiply both sides of (13.99) by a(z,,) and substitute for p(Xn+1|Zn+1) 
and p(Zn+41|Zn) using (13.75) and (13.76). Then we make use of (13.89), (13.90) 
and (13.91), together with (13.98), and after some manipulation we obtain 


n = Mn t+In (ing — Any) (13.100) 
Se a, (an = Pa) gr (13.101) 


ii 
Vv 


where we have defined 
Jn = VnA™ (Pn) (13.102) 


and we have made use of AV,, = P „J 1: Note that these recursions require that the 
forward pass be completed first so that the quantities yz,, and V,„ will be available 
for the backward pass. 

For the EM algorithm, we also require the pairwise posterior marginals, which 
can be obtained from (13.65) in the form 


ti eos ~~ 


€(Zn—1,Zn) = (Cn) @(2n—1) p(Xn|Zn) (Zn |Z—1) 8 (Zn) 
N (Zn—1|Mn—1» Wn—1)N (2n|AZn—1, TN (Xn|C2n, E)N (Zn| An, Vn) 
CnQi(Zn) ` 


(13.103) 


Substituting for &(Zn) using (13.84) and rearranging, we see that £(Zn—1, Zn) is a 
Gaussian with mean given with components y(Zn—1) and y(Zn), and a covariance 
between z,, and z,,_; given by 


cov|Zn,Zn—1] = Jn-1 Vn. (13.104) 
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13.3.2 Learning in LDS 


So far, we have considered the inference problem for linear dynamical systems, 
assuming that the model parameters 0 = {A, T, C, X, pt), Vo} are known. Next, we 
consider the determination of these parameters using maximum likelihood (Ghahra- 
mani and Hinton, 1996b). Because the model has latent variables, this can be ad- 
dressed using the EM algorithm, which was discussed in general terms in Chapter 9. 

We can derive the EM algorithm for the linear dynamical system as follows. Let 
us denote the estimated parameter values at some particular cycle of the algorithm 
by 0”, For these parameter values, we can run the inference algorithm to determine 
the posterior distribution of the latent variables p(Z|X, 0°"), or more precisely those 
local posterior marginals that are required in the M step. In particular, we shall 
require the following expectations 


E [Zn] = fn (13.105) 
Bente «| = InaVa+ Pel (13.106) 
Elat] = Vet Beh, (13.107) 


where we have used (13.104). 
Now we consider the complete-data log likelihood function, which is obtained 
by taking the logarithm of (13.6) and is therefore given by 


N 
In p(X, Z|0) = In p(Z1 |o, Vo) + X In p(2n|zn—1,4,T) 
n=2 
N 
+X mp(xnlzn, C, 5) (13.108) 


n=1 


in which we have made the dependence on the parameters explicit. We now take the 
expectation of the complete-data log likelihood with respect to the posterior distri- 
bution p(Z|X, 0°") which defines the function 


Q(0, 0°") = Eggo [In p(X, Z|) . (13.109) 


In the M step, this function is maximized with respect to the components of 0. 
Consider first the parameters pọ and Vo. If we substitute for p(zi|45, Vo) in 
(13.108) using (13.77), and then take the expectation with respect to Z, we obtain 


é 1 , 1 _ 
Q(0,0°") = = In|Vo| — Eggo 5 (ZA — Ho) V3 (zı — My) | + const 


where all terms not dependent on po or Vo have been absorbed into the additive 
constant. Maximization with respect to fg and Vo is easily performed by making 
use of the maximum likelihood solution for a Gaussian distribution discussed in 
Section 2.3.4, giving 
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ie = E[Z1] (13.110) 
Vom — E[z1z1 | — E |Z] E|z; |. (13.111) 


Similarly, to optimize A and I’, we substitute for p(Zn|Zn—1, A, T) in (13.108) 
using (13.75) giving 


N= 


gaye ii 


N 
1 
= Eggo ; Y n — Azp E (zA — A)| + const (13.112) 


n=2 


in which the constant comprises terms that are independent of A and T. Maximizing 


Exercise 13.33 with respect to these parameters then gives 
N N i 
AEN (>: E a (>: E [Zn =) (13.113) 
n=2 n=2 
ie z 
rer = OR >, { E [Zn Zy | = A™YE [Zn 12, | 


E [z,22_,] Ab” + AE [zy _127_,] (A2")? bs (13.114) 


Note that A™°” must be evaluated first, and the result can then be used to determine 
po. 

Finally, in order to determine the new values of C and X, we substitute for 
P(Xn|Zn, C, X) in (13.108) using (13.76) giving 


N 
Q(e,e°") = -7 h|] 
N 
—Ezjgoa 5 n — Czn) © (xn — Cz) | + const. 
Exercise 13.34 Maximizing with respect to C and © then gives 
N N = 
ow = z XnE 41) (>: E a) (13.115) 
n=1 

new __ new T 

» tae 5 ia — CVE [zn] x; 


-xn E [zn] C" + CE [znz] Cv}. (13.116) 
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We have approached parameter learning in the linear dynamical system using 
maximum likelihood. Inclusion of priors to give a MAP estimate is straightforward, 
and a fully Bayesian treatment can be found by applying the analytical approxima- 
tion techniques discussed in Chapter 10, though a detailed treatment is precluded 
here due to lack of space. 


13.3.3 Extensions of LDS 


As with the hidden Markov model, there is considerable interest in extending 
the basic linear dynamical system in order to increase its capabilities. Although the 
assumption of a linear-Gaussian model leads to efficient algorithms for inference 
and learning, it also implies that the marginal distribution of the observed variables 
is simply a Gaussian, which represents a significant limitation. One simple extension 
of the linear dynamical system is to use a Gaussian mixture as the initial distribution 
for zı. If this mixture has K components, then the forward recursion equations 
(13.85) will lead to a mixture of K Gaussians over each hidden variable z,,, and so 
the model is again tractable. 

For many applications, the Gaussian emission density is a poor approximation. 
If instead we try to use a mixture of K Gaussians as the emission density, then the 
posterior Q(z) will also be a mixture of K Gaussians. However, from (13.85) the 
posterior Q(z2) will comprise a mixture of K? Gaussians, and so on, with @(Zn) 
being given by a mixture of K” Gaussians. Thus the number of components grows 
exponentially with the length of the chain, and so this model is impractical. 

More generally, introducing transition or emission models that depart from the 
linear-Gaussian (or other exponential family) model leads to an intractable infer- 
ence problem. We can make deterministic approximations such as assumed den- 
sity filtering or expectation propagation, or we can make use of sampling methods, 
as discussed in Section 13.3.4. One widely used approach is to make a Gaussian 
approximation by linearizing around the mean of the predicted distribution, which 
gives rise to the extended Kalman filter (Zarchan and Musoff, 2005). 

As with hidden Markov models, we can develop interesting extensions of the ba- 
sic linear dynamical system by expanding its graphical representation. For example, 
the switching state space model (Ghahramani and Hinton, 1998) can be viewed as 
a combination of the hidden Markov model with a set of linear dynamical systems. 
The model has multiple Markov chains of continuous linear-Gaussian latent vari- 
ables, each of which is analogous to the latent chain of the linear dynamical system 
discussed earlier, together with a Markov chain of discrete variables of the form used 
in a hidden Markov model. The output at each time step is determined by stochas- 
tically choosing one of the continuous latent chains, using the state of the discrete 
latent variable as a switch, and then emitting an observation from the corresponding 
conditional output distribution. Exact inference in this model is intractable, but vari- 
ational methods lead to an efficient inference scheme involving forward-backward 
recursions along each of the continuous and discrete Markov chains independently. 
Note that, if we consider multiple chains of discrete latent variables, and use one as 
the switch to select from the remainder, we obtain an analogous model having only 
discrete latent variables known as the switching hidden Markov model. 
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13.3.4 Particle filters 


For dynamical systems which do not have a linear-Gaussian, for example, if 
they use a non-Gaussian emission density, we can turn to sampling methods in order 
to find a tractable inference algorithm. In particular, we can apply the sampling- 
importance-resampling formalism of Section 11.1.5 to obtain a sequential Monte 
Carlo algorithm known as the particle filter. 

Consider the class of distributions represented by the graphical model in Fig- 
ure 13.5, and suppose we are given the observed values X,, = (X1,...,Xņn) and 
we wish to draw L samples from the posterior distribution p(z,,|X,,). Using Bayes’ 
theorem, we have 


ELf@n)] = / f(@n)p(@n|Xn) dn 


EA Xn—1) dZn 
f F(n)p(%n\2n)P(@nlXn—1) dza 
J v0<olzn)DCenIXn—1) dam 


L 
aS ul Fe) (13.117) 


where {z\))} is a set of samples drawn from p(Zn|Xn—1) and we have made use of 
the conditional independence property p(Xp|Zn, Xn—1) = p(Xn|Zn), which follows 


from the graph in Figure 13.5. The sampling weights {wi 2 are defined by 


o 
M PXnlza ) (13.118) 


Wy, T 
Di 1 p(xnlzg”) 


where the same samples are used in the numerator as in the denominator. Thus the 
posterior distribution p(z,,|x,,) is represented by the set of samples {zt } together 


with the corresponding weights {wg ) }. Note that these weights satisfy 0 < w%®1 
and >>, we! = 

Because we wish to find a sequential sampling scheme, we shall suppose that 
a set of samples and weights have been obtained at time step n, and that we have 
subsequently observed the value of x,,,;, and we wish to find the weights and sam- 
ples at time step n + 1. We first sample from the distribution p(Zn+1|Xn). This is 
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straightforward since, again using Bayes’ theorem 
phensslXn) = | plansalens%Xn)Pl@n|Xn) dn 


= | r(ensaltn)PCGnI%n) dan 


= | plenssen)plen|x%n, Xn) dn 
| r62nsaltn)P(%nl2n)P(@n Xn) dz, 


J v0solen)oCenI%n—1) dz, 


X wO planla) (13.119) 
l 


where we have made use of the conditional independence properties 


PlZn+1|Zn; Xn) = P(Zn+1|Zn) (13.120) 
P(XnlZn,Xn-1) = P(Xn|Zn) (13.121) 


which follow from the application of the d-separation criterion to the graph in Fig- 
ure 13.5. The distribution given by (13.119) is a mixture distribution, and samples 
can be drawn by choosing a component l with probability given by the mixing coef- 
ficients w and then drawing a sample from the corresponding component. 

In summary, we can view each step of the particle filter algorithm as comprising 
two stages. At time step n, we have a sample representation of the posterior dis- 
tribution p(z,,|X,) expressed as samples {z\!)} with corresponding weights {w® }. 
This can be viewed as a mixture representation of the form (13.119). To obtain the 
corresponding representation for the next time step, we first draw L samples from 


the mixture distribution (13.119), and then for each sample we use the new obser- 


vation X,,+1 to evaluate the corresponding weights wo, x p(X%neilz0 ,). This is 


illustrated, for the case of a single variable z, in Figure 13.23. 

The particle filtering, or sequential Monte Carlo, approach has appeared in the 
literature under various names including the bootstrap filter (Gordon et al., 1993), 
survival of the fittest (Kanazawa et al., 1995), and the condensation algorithm (Isard 
and Blake, 1998). 


(x) FY = Use the technique of d-separation, discussed in Section 8.2, to verify 
that the Markov model shown in Figure 13.3 having N nodes in total satisfies the 
conditional independence properties (13.3) for n = 2,..., N. Similarly, show that 
a model described by the graph in Figure 13.4 in which there are N nodes in total 
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P(Xn41|2Zn41) 


P(Zn41|Xn41) 


Figure 13.23 Schematic illustration of the operation of the particle filter for a one-dimensional latent 


13.2 


13.3 


13.4 


space. At time step n, the posterior p(zn|xn) is represented as a mixture distribution, 
shown schematically as circles whose sizes are proportional to the weights w% . A set of 
L samples is then drawn from this distribution and the new weights wo, evaluated using 


P(Xn+1/20 11). 


satisfies the conditional independence properties 
P(Xn|X1; ey Rd) = (Xn |Xn—1,Xn—2) (13.122) 
for n = 3,...,N. 


(xx) Consider the joint probability distribution (13.2) corresponding to the directed 
graph of Figure 13.3. Using the sum and product rules of probability, verify that 
this joint distribution satisfies the conditional independence property (13.3) for n = 
2,...,N. Similarly, show that the second-order Markov model described by the 
joint distribution (13.4) satisfies the conditional independence property 


OX, Mayet yt) = P(Xn|Xn—-1; Xn—2) (13.123) 
for n = 3,..., N. 


(x) By using d-separation, show that the distribution p(x1,..., Xy ) of the observed 
data for the state space model represented by the directed graph in Figure 13.5 does 
not satisfy any conditional independence properties and hence does not exhibit the 
Markov property at any finite order. 


œ» EQ Consider a hidden Markov model in which the emission densities are 
represented by a parametric model p(x|z,w), such as a linear regression model or 
a neural network, in which w is a vector of adaptive parameters. Describe how the 
parameters w can be learned from data using maximum likelihood. 
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13.5 


13.6 


13.7 


13.8 


13.9 


13.10 


13.11 


13.12 


(xx) Verify the M-step equations (13.18) and (13.19) for the initial state probabili- 
ties and transition probability parameters of the hidden Markov model by maximiza- 
tion of the expected complete-data log likelihood function (13.17), using appropriate 
Lagrange multipliers to enforce the summation constraints on the components of 7 
and A. 


(x) Show that if any elements of the parameters m or A for a hidden Markov 
model are initially set to zero, then those elements will remain zero in all subsequent 
updates of the EM algorithm. 


(x) Consider a hidden Markov model with Gaussian emission densities. Show that 
maximization of the function Q(0, 0°") with respect to the mean and covariance 
parameters of the Gaussians gives rise to the M-step equations (13.20) and (13.21). 


(« x) FQ For a hidden Markov model having discrete observations governed by 
a multinomial distribution, show that the conditional distribution of the observations 
given the hidden variables is given by (13.22) and the corresponding M step equa- 
tions are given by (13.23). Write down the analogous equations for the conditional 
distribution and the M step equations for the case of a hidden Markov with multiple 
binary output variables each of which is governed by a Bernoulli conditional dis- 
tribution. Hint: refer to Sections 2.1 and 2.2 for a discussion of the corresponding 
maximum likelihood solutions for i.i.d. data if required. 


(x) EA = Use the d-separation criterion to verify that the conditional indepen- 
dence properties (13.24)-(13.31) are satisfied by the joint distribution for the hidden 
Markov model defined by (13.6). 


(xxx) By applying the sum and product rules of probability, verify that the condi- 
tional independence properties (13.24)-(13.31) are satisfied by the joint distribution 
for the hidden Markov model defined by (13.6). 


(xx) Starting from the expression (8.72) for the marginal distribution over the vari- 
ables of a factor in a factor graph, together with the results for the messages in the 
sum-product algorithm obtained in Section 13.2.3, derive the result (13.43) for the 
joint posterior distribution over two successive latent variables in a hidden Markov 
model. 


(xx) Suppose we wish to train a hidden Markov model by maximum likelihood 
using data that comprises R independent sequences of observations, which we de- 
note by X") where r = 1,..., R. Show that in the E step of the EM algorithm, 
we simply evaluate posterior probabilities for the latent variables by running the a 
and (3 recursions independently for each of the sequences. Also show that in the 
M step, the initial probability and transition probability parameters are re-estimated 


13.13 


13.14 


13.15 


13.16 
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using modified forms of (13.18 ) and (13.19) given by 


R 
ye 
r=1 


Tk = K (13.124) 
ey) 
r=1 j=1 
R N 
DD Elh ig Zaa) 
Ap = SS y (13.125) 


3 D 2 Elig a) 


r=1 l=1 n=2 


where, for notational convenience, we have assumed that the sequences are of the 
same length (the generalization to sequences of different lengths is straightforward). 
Similarly, show that the M-step equation for re-estimation of the means of Gaussian 
emission models is given by 


Hr = =E . (13.126) 


Note that the M-step equations for other emission model parameters and distributions 
take an analogous form. 


œx» EA Use the definition (8.64) of the messages passed from a factor node 
to a variable node in a factor graph, together with the expression (13.6) for the joint 
distribution in a hidden Markov model, to show that the definition (13.50) of the 
alpha message is the same as the definition (13.34). 


(xx) Use the definition (8.67) of the messages passed from a factor node to a 
variable node in a factor graph, together with the expression (13.6) for the joint 
distribution in a hidden Markov model, to show that the definition (13.52) of the 
beta message is the same as the definition (13.35). 


(xx) Use the expressions (13.33) and (13.43) for the marginals in a hidden Markov 
model to derive the corresponding results (13.64) and (13.65) expressed in terms of 
re-scaled variables. 


(xxx) In this exercise, we derive the forward message passing equation for the 
Viterbi algorithm directly from the expression (13.6) for the joint distribution. This 
involves maximizing over all of the hidden variables z1,...,z. By taking the log- 
arithm and then exchanging maximizations and summations, derive the recursion 
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13.17 


13.18 


13.19 


13.20 
13.21 


13.22 


13.23 


13.24 


13.25 


(13.68) where the quantities w(z,,) are defined by (13.70). Show that the initial 
condition for this recursion is given by (13.69). 


(x) [XY Show that the directed graph for the input-output hidden Markov model, 
given in Figure 13.18, can be expressed as a tree-structured factor graph of the form 
shown in Figure 13.15 and write down expressions for the initial factor h(z,) and 
for the general factor fn(Zn—1, Zn) where 2 < n < N. 


(xxx) Using the result of Exercise 13.17, derive the recursion equations, includ- 
ing the initial conditions, for the forward-backward algorithm for the input-output 
hidden Markov model shown in Figure 13.18. 


(x) (QQ The Kalman filter and smoother equations allow the posterior distribu- 
tions over individual latent variables, conditioned on all of the observed variables, 
to be found efficiently for linear dynamical systems. Show that the sequence of 
latent variable values obtained by maximizing each of these posterior distributions 
individually is the same as the most probable sequence of latent values. To do this, 
simply note that the joint distribution of all latent and observed variables in a linear 
dynamical system is Gaussian, and hence all conditionals and marginals will also be 
Gaussian, and then make use of the result (2.98). 


(xx) FQ Use the result (2.115) to prove (13.87). 


(xx) Use the results (2.115) and (2.116), together with the matrix identities (C.5) 
and (C.7), to derive the results (13.89), (13.90), and (13.91), where the Kalman gain 
matrix K, is defined by (13.92). 


(xx) FY Using (13.93), together with the definitions (13.76) and (13.77) and 
the result (2.115), derive (13.96). 


(xx) Using (13.93), together with the definitions (13.76) and (13.77) and the result 
(2.116), derive (13.94), (13.95) and (13.97). 


œx) FY Consider a generalization of (13.75) and (13.76) in which we include 
constant terms a and c in the Gaussian means, so that 


P(Zn|Zn—1) = N (Zn| AZn- + a,T) (13.127) 
P(Xn|Zn) = N(xn|Can + c, X). (13.128) 


Show that this extension can be re-case in the framework discussed in this chapter by 
defining a state vector z with an additional component fixed at unity, and then aug- 
menting the matrices A and C using extra columns corresponding to the parameters 
aand c. 


(xx) In this exercise, we show that when the Kalman filter equations are applied 
to independent observations, they reduce to the results given in Section 2.3 for the 
maximum likelihood solution for a single Gaussian distribution. Consider the prob- 
lem of finding the mean p of a single Gaussian random variable x, in which we are 
given a set of independent observations {x1,..., £n}. To model this we can use 


13.26 


13.27 


13.28 


13.29 


13.30 


13.31 


13.32 


13.33 
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a linear dynamical system governed by (13.75) and (13.76), with latent variables 
{z1,..-,2n}in which C becomes the identity matrix and where the transition prob- 
ability A = O because the observations are independent. Let the parameters mo 
and Vo of the initial state be denoted by 4o and o@, respectively, and suppose that 
= becomes c?. Write down the corresponding Kalman filter equations starting from 
the general results (13.89) and (13.90), together with (13.94) and (13.95). Show that 
these are equivalent to the results (2.141) and (2.142) obtained directly by consider- 
ing independent data. 


(xxx) Consider a special case of the linear dynamical system of Section 13.3 that is 
equivalent to probabilistic PCA, so that the transition matrix A = 0, the covariance 
T = I, and the noise covariance © = o7I. By making use of the matrix inversion 
identity (C.7) show that, if the emission density matrix C is denoted W, then the 
posterior distribution over the hidden states defined by (13.89) and (13.90) reduces 
to the result (12.42) for probabilistic PCA. 


œ QTY Consider a linear dynamical system of the form discussed in Sec- 
tion 13.3 in which the amplitude of the observation noise goes to zero, so that & = 0. 
Show that the posterior distribution for z,, has mean x, and zero variance. This 
accords with our intuition that if there is no noise, we should just use the current 
observation x,, to estimate the state variable z,, and ignore all previous observations. 


(xxx) Consider a special case of the linear dynamical system of Section 13.3 in 
which the state variable z,, is constrained to be equal to the previous state variable, 
which corresponds to A = I and T = 0. For simplicity, assume also that Vo — oo 
so that the initial conditions for z are unimportant, and the predictions are determined 
purely by the data. Use proof by induction to show that the posterior mean for state 
Zn is determined by the average of x,,...,X,. This corresponds to the intuitive 
result that if the state variable is constant, our best estimate is obtained by averaging 
the observations. 


(xxx) Starting from the backwards recursion equation (13.99), derive the RTS 
smoothing equations (13.100) and (13.101) for the Gaussian linear dynamical sys- 
tem. 


(xx) Starting from the result (13.65) for the pairwise posterior marginal in a state 
space model, derive the specific form (13.103) for the case of the Gaussian linear 
dynamical system. 


(xx) Starting from the result (13.103) and by substituting for a(z,,) using (13.84), 
verify the result (13.104) for the covariance between z,, and Z,_1. 


(« x) www | Verify the results (13.110) and (13.111) for the M-step equations for 
Ho and Vo in the linear dynamical system. 


(xx) Verify the results (13.113) and (13.114) for the M-step equations for A and T 
in the linear dynamical system. 
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13.34 (xx) Verify the results (13.115) and (13.116) for the M-step equations for C and X 
in the linear dynamical system. 
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In earlier chapters, we have explored a range of different models for solving classifi- 
cation and regression problems. It is often found that improved performance can be 
obtained by combining multiple models together in some way, instead of just using 
a single model in isolation. For instance, we might train L different models and then 
make predictions using the average of the predictions made by each model. Such 
combinations of models are sometimes called committees. In Section 14.2, we dis- 
cuss ways to apply the committee concept in practice, and we also give some insight 
into why it can sometimes be an effective procedure. 

One important variant of the committee method, known as boosting, involves 
training multiple models in sequence in which the error function used to train a par- 
ticular model depends on the performance of the previous models. This can produce 
substantial improvements in performance compared to the use of a single model and 
is discussed in Section 14.3. 

Instead of averaging the predictions of a set of models, an alternative form of 
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14.1. 


model combination is to select one of the models to make the prediction, in which 
the choice of model is a function of the input variables. Thus different models be- 
come responsible for making predictions in different regions of input space. One 
widely used framework of this kind is known as a decision tree in which the selec- 
tion process can be described as a sequence of binary selections corresponding to 
the traversal of a tree structure and is discussed in Section 14.4. In this case, the 
individual models are generally chosen to be very simple, and the overall flexibility 
of the model arises from the input-dependent selection process. Decision trees can 
be applied to both classification and regression problems. 

One limitation of decision trees is that the division of input space is based on 
hard splits in which only one model is responsible for making predictions for any 
given value of the input variables. The decision process can be softened by moving 
to a probabilistic framework for combining models, as discussed in Section 14.5. For 
example, if we have a set of K models for a conditional distribution p(t|x, k) where 
x is the input variable, t is the target variable, and k = 1,..., K indexes the model, 
then we can form a probabilistic mixture of the form 


K 
p(t|x) = So me(x)p(tlx, k) (14.1) 
k=1 


in which m(x) = p(k|x) represent the input-dependent mixing coefficients. Such 
models can be viewed as mixture distributions in which the component densities, as 
well as the mixing coefficients, are conditioned on the input variables and are known 
as mixtures of experts. They are closely related to the mixture density network model 
discussed in Section 5.6. 


Bayesian Model Averaging 


It is important to distinguish between model combination methods and Bayesian 
model averaging, as the two are often confused. To understand the difference, con- 
sider the example of density estimation using a mixture of Gaussians in which several 
Gaussian components are combined probabilistically. The model contains a binary 
latent variable z that indicates which component of the mixture is responsible for 
generating the corresponding data point. Thus the model is specified in terms of a 
joint distribution 

p(x, Z) (14.2) 


and the corresponding density over the observed variable x is obtained by marginal- 
izing over the latent variable 


p(x) = >" p(x, 2). (14.3) 
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In the case of our Gaussian mixture example, this leads to a distribution of the form 


K 
p(x) = X TN (|My, Be) (14.4) 


k=1 


with the usual interpretation of the symbols. This is an example of model combi- 
nation. For independent, identically distributed data, we can use (14.3) to write the 


marginal probability of a data set X = {x1,. . . , Xy } in the form 
N N 
p(X) = [[ ee) = II [Erez] . (14.5) 
n=1 n=1 L z, 


Thus we see that each observed data point x,, has a corresponding latent variable z,,. 

Now suppose we have several different models indexed by h = 1,...,H with 
prior probabilities p(h). For instance one model might be a mixture of Gaussians and 
another model might be a mixture of Cauchy distributions. The marginal distribution 
over the data set is given by 


A 
p(X) = X p(X|h)p(h). (14.6) 


h=1 


This is an example of Bayesian model averaging. The interpretation of this summa- 
tion over h is that just one model is responsible for generating the whole data set, 
and the probability distribution over h simply reflects our uncertainty as to which 
model that is. As the size of the data set increases, this uncertainty reduces, and 
the posterior probabilities p(h|X) become increasingly focussed on just one of the 
models. 

This highlights the key difference between Bayesian model averaging and model 
combination, because in Bayesian model averaging the whole data set is generated 
by a single model. By contrast, when we combine multiple models, as in (14.5), we 
see that different data points within the data set can potentially be generated from 
different values of the latent variable z and hence by different components. 

Although we have considered the marginal probability p(X), the same consid- 
erations apply for the predictive density p(x|X) or for conditional distributions such 
as p(t|x, X, T). 


Committees 


The simplest way to construct a committee is to average the predictions of a set of 
individual models. Such a procedure can be motivated from a frequentist perspective 
by considering the trade-off between bias and variance, which decomposes the er- 
ror due to a model into the bias component that arises from differences between the 
model and the true function to be predicted, and the variance component that repre- 
sents the sensitivity of the model to the individual data points. Recall from Figure 3.5 
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that when we trained multiple polynomials using the sinusoidal data, and then aver- 
aged the resulting functions, the contribution arising from the variance term tended to 
cancel, leading to improved predictions. When we averaged a set of low-bias mod- 
els (corresponding to higher order polynomials), we obtained accurate predictions 
for the underlying sinusoidal function from which the data were generated. 

In practice, of course, we have only a single data set, and so we have to find 
a way to introduce variability between the different models within the committee. 
One approach is to use bootstrap data sets, discussed in Section 1.2.3. Consider a 
regression problem in which we are trying to predict the value of a single continuous 
variable, and suppose we generate M bootstrap data sets and then use each to train 
a separate copy Ym(x) of a predictive model where m = 1,..., M. The committee 
prediction is given by 


M 
1 
ycom(*) = -7 2 Ym()- (14.7) 
mær 


This procedure is known as bootstrap aggregation or bagging (Breiman, 1996). 

Suppose the true regression function that we are trying to predict is given by 
h(x), so that the output of each of the models can be written as the true value plus 
an error in the form 


Ym(x) = A(x) + €n(X). (14.8) 
The average sum-of-squares error then takes the form 
Ex [{¥m (x) — h(x) }"] = Ex [em(x)”] (14.9) 


where E,|-] denotes a frequentist expectation with respect to the distribution of the 
input vector x. The average error made by the models acting individually is therefore 


M 
1 
Bw = 35 > Ex [em(x)?] - (14.10) 
mæl 


Similarly, the expected error from the committee (14.7) is given by 


v 2 
Ecom = Ex È > Ym(X) — Hoo} 


1 2 
= E faza] (14.11) 
oa 


If we assume that the errors have zero mean and are uncorrelated, so that 


Ex [en(x)] = 0 (14.12) 
Ex [Em(x)er(x)] = 0, ml (14.13) 
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then we obtain i 
Ecom = We: (14.14) 


This apparently dramatic result suggests that the average error of a model can be 
reduced by a factor of M simply by averaging M versions of the model. Unfortu- 
nately, it depends on the key assumption that the errors due to the individual models 
are uncorrelated. In practice, the errors are typically highly correlated, and the reduc- 
tion in overall error is generally small. It can, however, be shown that the expected 
committee error will not exceed the expected error of the constituent models, so that 
Ecom < Fay. In order to achieve more significant improvements, we turn to a more 
sophisticated technique for building committees, known as boosting. 


Boosting 


Boosting is a powerful technique for combining multiple ‘base’ classifiers to produce 
a form of committee whose performance can be significantly better than that of any 
of the base classifiers. Here we describe the most widely used form of boosting 
algorithm called AdaBoost, short for ‘adaptive boosting’, developed by Freund and 
Schapire (1996). Boosting can give good results even if the base classifiers have a 
performance that is only slightly better than random, and hence sometimes the base 
classifiers are known as weak learners. Originally designed for solving classification 
problems, boosting can also be extended to regression (Friedman, 2001). 

The principal difference between boosting and the committee methods such as 
bagging discussed above, is that the base classifiers are trained in sequence, and 
each base classifier is trained using a weighted form of the data set in which the 
weighting coefficient associated with each data point depends on the performance 
of the previous classifiers. In particular, points that are misclassified by one of the 
base classifiers are given greater weight when used to train the next classifier in 
the sequence. Once all the classifiers have been trained, their predictions are then 
combined through a weighted majority voting scheme, as illustrated schematically 
in Figure 14.1. 

Consider a two-class classification problem, in which the training data comprises 
input vectors X1, . . . , Xy along with corresponding binary target variables t1, ..., tN 
where tn € {—1,1}. Each data point is given an associated weighting parameter 
Wn, Which is initially set 1/N for all data points. We shall suppose that we have 
a procedure available for training a base classifier using weighted data to give a 
function y(x) € {—1,1}. At each stage of the algorithm, AdaBoost trains a new 
classifier using a data set in which the weighting coefficients are adjusted according 
to the performance of the previously trained classifier so as to give greater weight 
to the misclassified data points. Finally, when the desired number of base classifiers 
have been trained, they are combined to form a committee using coefficients that 
give different weight to different base classifiers. The precise form of the AdaBoost 
algorithm is given below. 
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Figure 14.1 Schematic illustration of the 
boosting framework. Each O CP) ao se? 2 Gu) 
base classifier ym (x) is trained 


on a weighted form of the train- 
ing set (blue arrows) in which eee 
the weights w\”) depend on 


the performance of the pre- 
vious base classifier ym—1(x) yi (x) yo(x) ym (x) 


(green arrows). Once all base 
classifiers have been trained, 
they are combined to give 


the final classifier Yar(x) (red 


arrows). M 
Yu (x) = sign 2 AmYm w) 
AdaBoost 


1. Initialize the data weighting coefficients {wn} by setting wi’) = 1/N for 
m= sag N: 


2. Frm = 1,..., M: 


(a) Fit a classifier ym(x) to the training data by minimizing the weighted 
error function 


N 
J = SL gan.) F te) (14.15) 


n=1 


where I(Ym(Xn) Æ tn) is the indicator function and equals 1 when 
Ym(Xn) É tn and 0 otherwise. 
(b) Evaluate the quantities 


N 
b> wE T(Ym (Xn) # tn) 


N 
(m) 


(14.16) 


Em 


and then use these to evaluate 


1 = 
am = Inf t), (14.17) 
Ern 


(3 


(c) Update the data weighting coefficients 


w™+D) = w™ exp {aml (Ym(Xn) Æ tn)} (14.18) 


n 
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3. Make predictions using the final model, which is given by 
M 
Yu(x) = sign (> aa) (14.19) 


We see that the first base classifier y;(x) is trained using weighting coeffi- 


cients w() that are all equal, which therefore corresponds to the usual procedure 
for training a single classifier. From (14.18), we see that in subsequent iterations 
the weighting coefficients we” are increased for data points that are misclassified 
and decreased for data points that are correctly classified. Successive classifiers are 
therefore forced to place greater emphasis on points that have been misclassified by 
previous classifiers, and data points that continue to be misclassified by successive 
classifiers receive ever greater weight. The quantities €m represent weighted mea- 
sures of the error rates of each of the base classifiers on the data set. We therefore 
see that the weighting coefficients am defined by (14.17) give greater weight to the 
more accurate classifiers when computing the overall output given by (14.19). 

The AdaBoost algorithm is illustrated in Figure 14.2, using a subset of 30 data 
points taken from the toy classification data set shown in Figure A.7. Here each base 
learners consists of a threshold on one of the input variables. This simple classifier 
corresponds to a form of decision tree known as a ‘decision stumps’, i.e., a deci- 
sion tree with a single node. Thus each base learner classifies an input according to 
whether one of the input features exceeds some threshold and therefore simply parti- 
tions the space into two regions separated by a linear decision surface that is parallel 
to one of the axes. 


14.3.1 Minimizing exponential error 


Boosting was originally motivated using statistical learning theory, leading to 
upper bounds on the generalization error. However, these bounds turn out to be too 
loose to have practical value, and the actual performance of boosting is much better 
than the bounds alone would suggest. Friedman et al. (2000) gave a different and 
very simple interpretation of boosting in terms of the sequential minimization of an 
exponential error function. 

Consider the exponential error function defined by 


N 
E =) exp {-tnfm(Xn)} (14.20) 


n=1 


where f(x) is a classifier defined in terms of a linear combination of base classifiers 
yi(x) of the form 


1 mMm 
m(x) = 3 > aryi(x) (14.21) 
=I 


and t,, € {—1, 1} are the training set target values. Our goal is to minimize Æ with 
respect to both the weighting coefficients a; and the parameters of the base classifiers 


y(x). 
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Illustration of boosting in which the base learners consist of simple thresholds applied to one or 


other of the axes. Each figure shows the number m of base learners trained so far, along with the decision 
boundary of the most recent base learner (dashed black line) and the combined decision boundary of the en- 
semble (solid green line). Each data point is depicted by a circle whose radius indicates the weight assigned to 
that data point when training the most recently added base learner. Thus, for instance, we see that points that 
are misclassified by the m = 1 base learner are given greater weight when training the m = 2 base learner. 


Instead of doing a global error function minimization, however, we shall sup- 
pose that the base classifiers y;(x),..., Ym—1(X) are fixed, as are their coefficients 
Q1,.-.,@m_—1, and so we are minimizing only with respect to a,,, and y,,(x). Sep- 
arating off the contribution from base classifier y,,(x), we can then write the error 
function in the form 


E 


II 
M 
o 

5 


{ —tn fm—1(Xn) — 5 tn mien %n) } 


I 
“M 
€ 
OD 

5 


y 1 
Cm) {-5tnntin(%n)} (14.22) 
n=1 2 

where the coefficients wh” = exp{—tnfm_—1(Xn)} can be viewed as constants 
because we are optimizing only am and ym(x). If we denote by Zm the set of 
data points that are correctly classified by y;,(x), and if we denote the remaining 
misclassified points by Mm, then we can in turn rewrite the error function in the 
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form 


E = em. > wl! m) 4. e%m/2 `> win 


nETm nEMm 


= (e am/2 _ e7 am/2) 1S a1 (xn) Aln) te ul 
(14.23) 


When we minimize this with respect to ym(x), we see that the second term is con- 
stant, and so this is equivalent to minimizing (14.15) because the overall multiplica- 
tive factor in front of the summation does not affect the location of the minimum. 
Similarly, minimizing with respect to &m, we obtain (14.17) in which em is defined 
by (14.16). 

From (14.22) we see that, having found am and ym(x), the weights on the data 
points are updated using 


1 
wht) = wl exp f- tanynt). (14.24) 


Making use of the fact that 


we see that the weights wl” 


wrt) = wi) exp(—Q,,/2) exp {amI (Ym(Xn) # tn)} ; (14.26) 


are updated at the next iteration using 


Because the term exp(—a,,,/2) is independent of n, we see that it weights all data 
points by the same factor and so can be discarded. Thus we obtain (14.18). 

Finally, once all the base classifiers are trained, new data points are classified by 
evaluating the sign of the combined function defined according to (14.21). Because 
the factor of 1/2 does not affect the sign it can be omitted, giving (14.19). 


14.3.2 Error functions for boosting 


The exponential error function that is minimized by the AdaBoost algorithm 
differs from those considered in previous chapters. To gain some insight into the 
nature of the exponential error function, we first consider the expected error given 
by 


Bes lexr{-—tw00}] = | otto oaoa 4427) 


If we perform a variational minimization with respect to all possible functions y(x), 


we obtain i ( Lx) 
_ p(t = 1|x 
y(x) = 5 In [z Zi } (14.28) 
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Figure 14.3 Plot of the exponential (green) and E(z) 
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rescaled cross-entropy (red) error 
functions along with the hinge er- 
ror (blue) used in support vector 
machines, and the misclassification 
error (black). Note that for large 
negative values of z = ty(x), the 
cross-entropy gives a linearly in- 
creasing penalty, whereas the expo- 
nential loss gives an exponentially in- 
creasing penalty. 


—2 —1 0 1 2 


which is half the log-odds. Thus the AdaBoost algorithm is seeking the best approx- 
imation to the log odds ratio, within the space of functions represented by the linear 
combination of base classifiers, subject to the constrained minimization resulting 
from the sequential optimization strategy. This result motivates the use of the sign 
function in (14.19) to arrive at the final classification decision. 

We have already seen that the minimizer y(x) of the cross-entropy error (4.90) 
for two-class classification is given by the posterior class probability. In the case 
of a target variable t € {—1,1}, we have seen that the error function is given by 
In(1 + exp(—yt)). This is compared with the exponential error function in Fig- 
ure 14.3, where we have divided the cross-entropy error by a constant factor In(2) 
so that it passes through the point (0,1) for ease of comparison. We see that both 
can be seen as continuous approximations to the ideal misclassification error func- 
tion. An advantage of the exponential error is that its sequential minimization leads 
to the simple AdaBoost scheme. One drawback, however, is that it penalizes large 
negative values of ty(x) much more strongly than cross-entropy. In particular, we 
see that for large negative values of ty, the cross-entropy grows linearly with |ty 
whereas the exponential error function grows exponentially with |ty|. Thus the ex- 
ponential error function will be much less robust to outliers or misclassified data 
points. Another important difference between cross-entropy and the exponential er- 
ror function is that the latter cannot be interpreted as the log likelihood function of 
any well-defined probabilistic model. Furthermore, the exponential error does not 
generalize to classification problems having K > 2 classes, again in contrast to the 
cross-entropy for a probabilistic model, which is easily generalized to give (4.108). 

The interpretation of boosting as the sequential optimization of an additive model 
under an exponential error (Friedman et al., 2000) opens the door to a wide range 
of boosting-like algorithms, including multiclass extensions, by altering the choice 
of error function. It also motivates the extension to regression problems (Friedman, 
2001). If we consider a sum-of-squares error function for regression, then sequential 
minimization of an additive model of the form (14.21) simply involves fitting each 
new base classifier to the residual errors tn — fim—1(Xn) from the previous model. As 
we have noted, however, the sum-of-squares error is not robust to outliers, and this 
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Comparison of the squared error 
(green) with the absolute error (red) 
showing how the latter places much 
less emphasis on large errors and 
hence is more robust to outliers and 
mislabelled data points. 


E(2) 


—1 0 1 Z 


can be addressed by basing the boosting algorithm on the absolute deviation |y — t| 
instead. These two error functions are compared in Figure 14.4. 


Tree-based Models 


There are various simple, but widely used, models that work by partitioning the 
input space into cuboid regions, whose edges are aligned with the axes, and then 
assigning a simple model (for example, a constant) to each region. They can be 
viewed as a model combination method in which only one model is responsible 
for making predictions at any given point in input space. The process of selecting 
a specific model, given a new input x, can be described by a sequential decision 
making process corresponding to the traversal of a binary tree (one that splits into 
two branches at each node). Here we focus on a particular tree-based framework 
called classification and regression trees, or CART (Breiman et al., 1984), although 
there are many other variants going by such names as ID3 and C4.5 (Quinlan, 1986; 
Quinlan, 1993). 

Figure 14.5 shows an illustration of a recursive binary partitioning of the input 
space, along with the corresponding tree structure. In this example, the first step 


Illustration of a two-dimensional in- 72 
put space that has been partitioned 
into five regions using axis-aligned 
boundaries. 
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Figure 14.6 Binary tree corresponding to the par- 


Exercise 14.10 


titioning of input space shown in Fig- 
ure 14.5. 


A B C D E 


divides the whole of the input space into two regions according to whether zı < 0; 
or x; > 6; where 6, is a parameter of the model. This creates two subregions, each 
of which can then be subdivided independently. For instance, the region x, < 6; 
is further subdivided according to whether ry < 62 or zə > Oo, giving rise to the 
regions denoted A and B. The recursive subdivision can be described by the traversal 
of the binary tree shown in Figure 14.6. For any new input x, we determine which 
region it falls into by starting at the top of the tree at the root node and following 
a path down to a specific leaf node according to the decision criteria at each node. 
Note that such decision trees are not probabilistic graphical models. 

Within each region, there is a separate model to predict the target variable. For 
instance, in regression we might simply predict a constant over each region, or in 
classification we might assign each region to a specific class. A key property of tree- 
based models, which makes them popular in fields such as medical diagnosis, for 
example, is that they are readily interpretable by humans because they correspond 
to a sequence of binary decisions applied to the individual input variables. For in- 
stance, to predict a patient’s disease, we might first ask “is their temperature greater 
than some threshold?”. If the answer is yes, then we might next ask “is their blood 
pressure less than some threshold?”. Each leaf of the tree is then associated with a 
specific diagnosis. 

In order to learn such a model from a training set, we have to determine the 
structure of the tree, including which input variable is chosen at each node to form 
the split criterion as well as the value of the threshold parameter 0; for the split. We 
also have to determine the values of the predictive variable within each region. 

Consider first a regression problem in which the goal is to predict a single target 


variable t from a D-dimensional vector x = (x1,...,2p)* of input variables. The 
training data consists of input vectors {x,,...,x.} along with the corresponding 
continuous labels {t1, . . . , ty }. If the partitioning of the input space is given, and we 


minimize the sum-of-squares error function, then the optimal value of the predictive 
variable within any given region is just given by the average of the values of tn for 
those data points that fall in that region. 

Now consider how to determine the structure of the decision tree. Even for a 
fixed number of nodes in the tree, the problem of determining the optimal structure 
(including choice of input variable for each split as well as the corresponding thresh- 
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olds) to minimize the sum-of-squares error is usually computationally infeasible due 
to the combinatorially large number of possible solutions. Instead, a greedy opti- 
mization is generally done by starting with a single root node, corresponding to the 
whole input space, and then growing the tree by adding nodes one at a time. At each 
step there will be some number of candidate regions in input space that can be split, 
corresponding to the addition of a pair of leaf nodes to the existing tree. For each 
of these, there is a choice of which of the D input variables to split, as well as the 
value of the threshold. The joint optimization of the choice of region to split, and the 
choice of input variable and threshold, can be done efficiently by exhaustive search 
noting that, for a given choice of split variable and threshold, the optimal choice of 
predictive variable is given by the local average of the data, as noted earlier. This 
is repeated for all possible choices of variable to be split, and the one that gives the 
smallest residual sum-of-squares error is retained. 

Given a greedy strategy for growing the tree, there remains the issue of when 
to stop adding nodes. A simple approach would be to stop when the reduction in 
residual error falls below some threshold. However, it is found empirically that often 
none of the available splits produces a significant reduction in error, and yet after 
several more splits a substantial error reduction is found. For this reason, it is com- 
mon practice to grow a large tree, using a stopping criterion based on the number 
of data points associated with the leaf nodes, and then prune back the resulting tree. 
The pruning is based on a criterion that balances residual error against a measure of 
model complexity. If we denote the starting tree for pruning by To, then we define 
T C To to be a subtree of To if it can be obtained by pruning nodes from To (in 
other words, by collapsing internal nodes by combining the corresponding regions). 
Suppose the leaf nodes are indexed by 7 = 1, . . . , |T|, with leaf node 7 representing 
a region R- of input space having N, data points, and |T| denoting the total number 
of leaf nodes. The optimal prediction for region R is then given by 


1 
Ue =e DE (14.29) 
Xn ER? 


and the corresponding contribution to the residual sum-of-squares is then 


Q(T) = So {azy (14.30) 


XnER, 
The pruning criterion is then given by 


IT 


O(T) = 5° QAT) +XT| (14.31) 
r=1 


The regularization parameter A determines the trade-off between the overall residual 
sum-of-squares error and the complexity of the model as measured by the number 
|T| of leaf nodes, and its value is chosen by cross-validation. 

For classification problems, the process of growing and pruning the tree is sim- 
ilar, except that the sum-of-squares error is replaced by a more appropriate measure 
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of performance. If we define p,; to be the proportion of data points in region R+ 
assigned to class k, where k = 1,..., A, then two commonly used choices are the 
cross-entropy 


K 
a=) palita (14.32) 
k=1 
and the Gini index "e 
Q(T) =J pre = pri): (14.33) 
k=1 


These both vanish for px = 0 and p,, = 1 and have a maximum at pyx = 0.5. They 
encourage the formation of regions in which a high proportion of the data points are 
assigned to one class. The cross entropy and the Gini index are better measures than 
the misclassification rate for growing the tree because they are more sensitive to the 
node probabilities. Also, unlike misclassification rate, they are differentiable and 
hence better suited to gradient based optimization methods. For subsequent pruning 
of the tree, the misclassification rate is generally used. 

The human interpretability of a tree model such as CART is often seen as its 
major strength. However, in practice it is found that the particular tree structure that 
is learned is very sensitive to the details of the data set, so that a small change to the 
training data can result in a very different set of splits (Hastie et al., 2001). 

There are other problems with tree-based methods of the kind considered in 
this section. One is that the splits are aligned with the axes of the feature space, 
which may be very suboptimal. For instance, to separate two classes whose optimal 
decision boundary runs at 45 degrees to the axes would need a large number of 
axis-parallel splits of the input space as compared to a single non-axis-aligned split. 
Furthermore, the splits in a decision tree are hard, so that each region of input space 
is associated with one, and only one, leaf node model. The last issue is particularly 
problematic in regression where we are typically aiming to model smooth functions, 
and yet the tree model produces piecewise-constant predictions with discontinuities 
at the split boundaries. 


Conditional Mixture Models 


We have seen that standard decision trees are restricted by hard, axis-aligned splits of 
the input space. These constraints can be relaxed, at the expense of interpretability, 
by allowing soft, probabilistic splits that can be functions of all of the input variables, 
not just one of them at a time. If we also give the leaf models a probabilistic inter- 
pretation, we arrive at a fully probabilistic tree-based model called the hierarchical 
mixture of experts, which we consider in Section 14.5.3. 

An alternative way to motivate the hierarchical mixture of experts model is to 
start with a standard probabilistic mixtures of unconditional density models such as 
Gaussians and replace the component densities with conditional distributions. Here 
we consider mixtures of linear regression models (Section 14.5.1) and mixtures of 
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logistic regression models (Section 14.5.2). In the simplest case, the mixing coeffi- 
cients are independent of the input variables. If we make a further generalization to 
allow the mixing coefficients also to depend on the inputs then we obtain a mixture 
of experts model. Finally, if we allow each component in the mixture model to be 
itself a mixture of experts model, then we obtain a hierarchical mixture of experts. 


14.5.1 Mixtures of linear regression models 


One of the many advantages of giving a probabilistic interpretation to the lin- 
ear regression model is that it can then be used as a component in more complex 
probabilistic models. This can be done, for instance, by viewing the conditional 
distribution representing the linear regression model as a node in a directed prob- 
abilistic graph. Here we consider a simple example corresponding to a mixture of 
linear regression models, which represents a straightforward extension of the Gaus- 
sian mixture model discussed in Section 9.2 to the case of conditional Gaussian 
distributions. 

We therefore consider K linear regression models, each governed by its own 
weight parameter wọ. In many applications, it will be appropriate to use a common 
noise variance, governed by a precision parameter /3, for all K components, and this 
is the case we consider here. We will once again restrict attention to a single target 
variable t, though the extension to multiple outputs is straightforward. If we denote 
the mixing coefficients by 7k, then the mixture distribution can be written 


p(t|@) = Sm (tlw Eb, 6° 1) (14.34) 


k=1 


where @ denotes the set of all adaptive parameters in the model, namely W = {wx}, 
aw = {Tk}, and 8. The log likelihood function for this model, given a data set of 
observations {@,,, tn}, then takes the form 


In p(t|@) = Sai (Save Iwen, 67 ») (14.35) 
n=l 


where t = (t4, ..., ty)" denotes the vector of target variables. 

In order to maximize this likelihood function, we can once again appeal to the 
EM algorithm, which will turn out to be a simple extension of the EM algorithm for 
unconditional Gaussian mixtures of Section 9.2. We can therefore build on our expe- 
rience with the unconditional mixture and introduce a set Z = {z,,} of binary latent 
variables where z,,, € {0,1} in which, for each data point n, all of the elements 
k = 1,..., are zero except for a single value of 1 indicating which component 
of the mixture was responsible for generating that data point. The joint distribution 
over latent and observed variables can be represented by the graphical model shown 
in Figure 14.7. 

The complete-data ae likelihood function then takes the form 


In p(t, Z|) = SY {aN (tawi pn 87}. (14.36) 
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Figure 14.7 Probabilistic directed graph representing a mixture of 


Exercise 14.14 


linear regression models, defined by (14.35). m & 


Ww 


The EM algorithm begins by first choosing an initial value 6°" for the model param- 
eters. In the E step, these parameter values are then used to evaluate the posterior 
probabilities, or responsibilities, of each component k for every data point n given 
by 

TEN (tr |WE Pn: 67t) 
a TiN (trlw; Pn, B-*) 
The responsibilities are then used to determine the expectation, with respect to the 


posterior distribution p(Z|t, 0°"), of the complete-data log likelihood, which takes 
the form 


nk = Elznx] = piklo, 0°") = (14.37) 


N K 
Q(0,0°"") = Ez [ln p(t, Z|6)] po rink {Inte + MN (tn|Wi Gn, 67)} - 
1 k=1 


In the M step, we maximize the function Q(0, 0°") with respect to 0, keeping the 
Ynk fixed. For the optimization with respect to the mixing coefficients m we need 
to take account of the constraint 5 Tk = 1, which can be done with the aid of a 
Lagrange multiplier, leading to an M-step re-estimation equation for 7;, in the form 


1 
T= > >, ‘nk: (14.38) 


Note that this has exactly the same form as the corresponding result for a simple 
mixture of unconditional Gaussians given by (9.22). 

Next consider the maximization with respect to the parameter vector w% of the 
k* linear regression model. Substituting for the Gaussian distribution, we see that 
the function Q(6, 0%), as a function of the parameter vector wx, takes the form 


0,0%?) = = Ink {-5¢ = »-wie,)’} + const (14.39) 


where the constant term includes the contributions from other weight vectors w; for 
j # k. Note that the quantity we are maximizing is similar to the (negative of the) 
standard sum-of-squares error (3.12) for a single linear regression model, but with 
the inclusion of the responsibilities y,,,. This represents a weighted least squares 
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problem, in which the term corresponding to the n*™™ data point carries a weighting 
coefficient given by 8 nx, which could be interpreted as an effective precision for 
each data point. We see that each component linear regression model in the mixture, 
governed by its own parameter vector wx, is fitted separately to the whole data set in 
the M step, but with each data point n weighted by the responsibility Yng that model 
k takes for that data point. Setting the derivative of (14.39) with respect to w;, equal 
to zero gives 


N 
0= So nk (tn = wen) Pn (14.40) 
n=1 


which we can write in matrix notation as 
0 = 6'R,(t— Pw) (14.41) 


where Ry = diag(Ynķ) is a diagonal matrix of size N x N. Solving for wp, we 
obtain = 
wk = (®'R,&) PTR. (14.42) 


This represents a set of modified normal equations corresponding to the weighted 
least squares problem, of the same form as (4.99) found in the context of logistic 
regression. Note that after each E step, the matrix Rẹ will change and so we will 
have to solve the normal equations afresh in the subsequent M step. 

Finally, we maximize Q(0, 0%) with respect to 8. Keeping only terms that 
depend on £, the function Q(@, 0°") can be written 


N K 1 B 
o 2 
Q(6,0°") = md In 8 ; (tn — WE Pn) fi (14.43) 
1 
Setting the derivative with respect to 8 equal to zero, and rearranging, we obtain the 
M-step equation for 8 in the form 


E en ae 
B= ada (tn — Win) - (14.44) 


n=1 k=1 


In Figure 14.8, we illustrate this EM algorithm using the simple example of 
fitting a mixture of two straight lines to a data set having one input variable x and 
one target variable t. The predictive density (14.34) is plotted in Figure 14.9 using 
the converged parameter values obtained from the EM algorithm, corresponding to 
the right-hand plot in Figure 14.8. Also shown in this figure is the result of fitting 
a single linear regression model, which gives a unimodal predictive density. We see 
that the mixture model gives a much better representation of the data distribution, 
and this is reflected in the higher likelihood value. However, the mixture model 
also assigns significant probability mass to regions where there is no data because its 
predictive distribution is bimodal for all values of x. This problem can be resolved by 
extending the model to allow the mixture coefficients themselves to be functions of 
x, leading to models such as the mixture density networks discussed in Section 5.6, 
and hierarchical mixture of experts discussed in Section 14.5.3. 
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Figure 14.8 Example of a synthetic data set, shown by the green points, having one input variable x and one 
target variable t, together with a mixture of two linear regression models whose mean functions y(x, wz), where 
k € {1,2}, are shown by the blue and red lines. The upper three plots show the initial configuration (left), the 
result of running 30 iterations of EM (centre), and the result after 50 iterations of EM (right). Here @ was initialized 
to the reciprocal of the true variance of the set of target values. The lower three plots show the corresponding 
responsibilities plotted as a vertical line for each data point in which the length of the blue segment gives the 
posterior probability of the blue line for that data point (and similarly for the red segment). 


14.5.2 Mixtures of logistic models 


Because the logistic regression model defines a conditional distribution for the 
target variable, given the input vector, it is straightforward to use it as the component 
distribution in a mixture model, thereby giving rise to a richer family of conditional 
distributions compared to a single logistic regression model. This example involves 
a straightforward combination of ideas encountered in earlier sections of the book 
and will help consolidate these for the reader. 

The conditional distribution of the target variable, for a probabilistic mixture of 
K logistic regression models, is given by 


D(t|?, 0) = Smut gg (14.45) 


where @ is the feature vector, yk = o (we ¢) is the output of component k, and 0 
denotes the adjustable parameters namely {7;,} and {wx}. 
Now suppose we are given a data set {@,,,t,}. The corresponding likelihood 
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Figure 14.9 The left plot shows the predictive conditional density corresponding to the converged solution in 
Figure 14.8. This gives a log likelihood value of —3.0. A vertical slice through one of these plots at a particular 
value of x represents the corresponding conditional distribution p(t|x), which we see is bimodal. The plot on the 
right shows the predictive density for a single linear regression model fitted to the same data set using maximum 
likelihood. This model has a smaller log likelihood of —27.6. 


function is then given by 


N K 
p(t|@) = I e mky [1 — mal) (14.46) 


where ynk = o(wE Q) and t = (tı,..., tn)". We can maximize this likelihood 
function iteratively by making use of the EM algorithm. This involves introducing 
latent variables z,,, that correspond to a l-of-K coded binary indicator variable for 
each data point n. The an likelihood function is then given by 


p(t, Z|0) = Il Il {rry [1 yn] 7 b (14.47) 


n=1k=1 


where Z is the matrix of latent variables with elements zng. We initialize the EM 
algorithm by choosing an initial value 0°" for the model parameters. In the E step, 
we then use these parameter values to evaluate the posterior probabilities of the com- 
ponents k for each data point n, which are given by 


meyer, [L — Yne] 


Dy TiYn [1 — Ying 


These responsibilities are then used to find the expected complete-data log likelihood 
as a function of 0, given by 


Q(9, es = Ez [In p(t, Z|)| 


nk SE eae| = plk|on, 0%) = (14.48) 


aa i 


- > 1 {Int + tr In ynk + (1— tn)ln(1— ynk)}. (14.49) 
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Exercise 14.16 


The M step involves maximization of this function with respect to 0, keeping 0°, 
and hence nx, fixed. Maximization with respect to 7; can be done in the usual way, 
with a Lagrange multiplier to enforce the summation constraint $`, 7, = 1, giving 
the familiar result 


1 
T= >, ‘nk: (14.50) 


To determine the {w+}, we note that the Q(@, 0°") function comprises a sum 
over terms indexed by k each of which depends only on one of the vectors w;, so 
that the different vectors are decoupled in the M step of the EM algorithm. In other 
words, the different components interact only via the responsibilities, which are fixed 
during the M step. Note that the M step does not have a closed-form solution and 
must be solved iteratively using, for instance, the iterative reweighted least squares 
(ARLS) algorithm. The gradient and the Hessian for the vector w;, are given by 


ViQ = Slt ~ Ynk) Pn (14.51) 
H, = -VEVO = 3 Yaatinall — Yas) oad? (14.52) 
n=1 


where V;, denotes the gradient with respect to wg. For fixed yz, these are indepen- 
dent of {w; } for j #4 k and so we can solve for each w;, separately using the IRLS 
algorithm. Thus the M-step equations for component k correspond simply to fitting 
a single logistic regression model to a weighted data set in which data point n carries 
a weight Yng. Figure 14.10 shows an example of the mixture of logistic regression 
models applied to a simple classification problem. The extension of this model to a 
mixture of softmax models for more than two classes is straightforward. 


14.5.3 Mixtures of experts 


In Section 14.5.1, we considered a mixture of linear regression models, and in 
Section 14.5.2 we discussed the analogous mixture of linear classifiers. Although 
these simple mixtures extend the flexibility of linear models to include more com- 
plex (e.g., multimodal) predictive distributions, they are still very limited. We can 
further increase the capability of such models by allowing the mixing coefficients 
themselves to be functions of the input variable, so that 


p(t|x) = Ynt x)pp(t|x). (14.53) 


This is known as a mixture of experts model (Jacobs et al., 1991) in which the mix- 
ing coefficients m(x) are known as gating functions and the individual component 
densities p;(t|x) are called experts. The notion behind the terminology is that differ- 
ent components can model the distribution in different regions of input space (they 
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Figure 14.10 Illustration of a mixture of logistic regression models. The left plot shows data points drawn 
from two classes denoted red and blue, in which the background colour (which varies from pure red to pure blue) 
denotes the true probability of the class label. The centre plot shows the result of fitting a single logistic regression 
model using maximum likelihood, in which the background colour denotes the corresponding probability of the 
class label. Because the colour is a near-uniform purple, we see that the model assigns a probability of around 
0.5 to each of the classes over most of input space. The right plot shows the result of fitting a mixture of two 
logistic regression models, which now gives much higher probability to the correct labels for many of the points 
in the blue class. 


are ‘experts’? at making predictions in their own regions), and the gating functions 
determine which components are dominant in which region. 

The gating functions m(x) must satisfy the usual constraints for mixing co- 
efficients, namely 0 < m(x) < 1 and $`, m(x) = 1. They can therefore be 
represented, for example, by linear softmax models of the form (4.104) and (4.105). 
If the experts are also linear (regression or classification) models, then the whole 
model can be fitted efficiently using the EM algorithm, with iterative reweighted 
least squares being employed in the M step (Jordan and Jacobs, 1994). 

Such a model still has significant limitations due to the use of linear models 
for the gating and expert functions. A much more flexible model is obtained by 
using a multilevel gating function to give the hierarchical mixture of experts, or 
HME model (Jordan and Jacobs, 1994). To understand the structure of this model, 
imagine a mixture distribution in which each component in the mixture is itself a 
mixture distribution. For simple unconditional mixtures, this hierarchical mixture is 

Exercise 14.17 trivially equivalent to a single flat mixture distribution. However, when the mixing 
coefficients are input dependent, this hierarchical model becomes nontrivial. The 
HME model can also be viewed as a probabilistic version of decision trees discussed 
in Section 14.4 and can again be trained efficiently by maximum likelihood using an 

Section 4.3.3 EM algorithm with IRLS in the M step. A Bayesian treatment of the HME has been 
given by Bishop and Svensén (2003) based on variational inference. 

We shall not discuss the HME in detail here. However, it is worth pointing out 
the close connection with the mixture density network discussed in Section 5.6. The 
principal advantage of the mixtures of experts model is that it can be optimized by 
EM in which the M step for each mixture component and gating model involves 
a convex optimization (although the overall optimization is nonconvex). By con- 
trast, the advantage of the mixture density network approach is that the component 
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Exercises 
14.1 


14.2 


14.3 


14.4 


14.5 


densities and the mixing coefficients share the hidden units of the neural network. 
Furthermore, in the mixture density network, the splits of the input space are further 
relaxed compared to the hierarchical mixture of experts in that they are not only soft, 
and not constrained to be axis aligned, but they can also be nonlinear. 


œ» [QTY Consider a set models of the form p(t|x, Zn, On, h) in which x is the 
input vector, t is the target vector, h indexes the different models, z, is a latent vari- 
able for model h, and 0, is the set of parameters for model h. Suppose the models 
have prior probabilities p(h) and that we are given a training set X = {x1,...,xv} 
and T = {t,...,t.}. Write down the formulae needed to evaluate the predic- 
tive distribution p(t|x, X, T) in which the latent variables and the model index are 
marginalized out. Use these formulae to highlight the difference between Bayesian 
averaging of different models and the use of latent variables within a single model. 


(x) The expected sum-of-squares error Eay for a simple committee model can 
be defined by (14.10), and the expected error of the committee itself is given by 
(14.11). Assuming that the individual errors satisfy (14.12) and (14.13), derive the 
result (14.14). 


(x) FY By making use of Jensen’s inequality (1.115), for the special case of 
the convex function f(x) = x7, show that the average expected sum-of-squares 
error E’ay of the members of a simple committee model, given by (14.10), and the 
expected error Ecom of the committee itself, given by (14.11), satisfy 


Ecom <S Eav. (14.54) 
(xx) By making use of Jensen’s in equality (1.115), show that the result (14.54) 


derived in the previous exercise hods for any error function E(y), not just sum-of- 
squares, provided it is a convex function of y. 


œ» FQ Consider a committee in which we allow unequal weighting of the 
constituent models, so that 


M 
Ycom(X) = 5 AmYm(X). (14.55) 
m=1 


In order to ensure that the predictions ycom(x) remain within sensible limits, sup- 
pose that we require that they be bounded at each value of x by the minimum and 
maximum values given by any of the members of the committee, so that 


Ymin(X) < Ycom(X) < Ymax( X): (14.56) 


Show that a necessary and sufficient condition for this constraint is that the coeffi- 
cients a, satisfy 


am > 0, 5 Am =l. (14.57) 
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œ FY By differentiating the error function (14.23) with respect to &m, show 
that the parameters am in the AdaBoost algorithm are updated using (14.17) in 
which em is defined by (14.16). 


(x) By making a variational minimization of the expected exponential error function 
given by (14.27) with respect to all possible functions y(x), show that the minimizing 
function is given by (14.28). 


(x) Show that the exponential error function (14.20), which is minimized by the 
AdaBoost algorithm, does not correspond to the log likelihood of any well-behaved 
probabilistic model. This can be done by showing that the corresponding conditional 
distribution p(t|x) cannot be correctly normalized. 


œ EX Show that the sequential minimization of the sum-of-squares error func- 
tion for an additive model of the form (14.21) in the style of boosting simply involves 
fitting each new base classifier to the residual errors tn — fm—1 (Xn ) from the previous 
model. 


(x) Verify that if we minimize the sum-of-squares error between a set of training 
values {t,,} and a single predictive value t, then the optimal solution for t is given 
by the mean of the {tn }. 


(xx) Consider a data set comprising 400 data points from class C, and 400 data 
points from class C2. Suppose that a tree model A splits these into (300, 100) at 
the first leaf node and (100, 300) at the second leaf node, where (n, m) denotes that 
n points are assigned to Cı and m points are assigned to C2. Similarly, suppose 
that a second tree model B splits them into (200,400) and (200,0). Evaluate the 
misclassification rates for the two trees and hence show that they are equal. Similarly, 
evaluate the cross-entropy (14.32) and Gini index (14.33) for the two trees and show 
that they are both lower for tree B than for tree A. 


(xx) Extend the results of Section 14.5.1 for a mixture of linear regression models 
to the case of multiple target values described by a vector t. To do this, make use of 
the results of Section 3.1.5. 


œ FY Verify that the complete-data log likelihood function for the mixture of 
linear regression models is given by (14.36). 


(x) Use the technique of Lagrange multipliers (Appendix E) to show that the M-step 
re-estimation equation for the mixing coefficients in the mixture of linear regression 
models trained by maximum likelihood EM is given by (14.38). 


œ EX We have already noted that if we use a squared loss function in a regres- 
sion problem, the corresponding optimal prediction of the target variable for a new 
input vector is given by the conditional mean of the predictive distribution. Show 
that the conditional mean for the mixture of linear regression models discussed in 
Section 14.5.1 is given by a linear combination of the means of each component dis- 
tribution. Note that if the conditional distribution of the target data is multimodal, 
the conditional mean can give poor predictions. 
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(xxx) Extend the logistic regression mixture model of Section 14.5.2 to a mixture 
of softmax classifiers representing C > 2 classes. Write down the EM algorithm for 
determining the parameters of this model through maximum likelihood. 


œ» [QTY Consider a mixture model for a conditional distribution p(t|x) of the 
form 


K 
p(tlx) = X weve (thx) (14.58) 
k=1 


in which each mixture component 7 (t|x) is itself a mixture model. Show that this 
two-level hierarchical mixture is equivalent to a conventional single-level mixture 
model. Now suppose that the mixing coefficients in both levels of such a hierar- 
chical model are arbitrary functions of x. Again, show that this hierarchical model 
is again equivalent to a single-level model with x-dependent mixing coefficients. 
Finally, consider the case in which the mixing coefficients at both levels of the hi- 
erarchical mixture are constrained to be linear classification (logistic or softmax) 
models. Show that the hierarchical mixture cannot in general be represented by a 
single-level mixture having linear classification models for the mixing coefficients. 
Hint: to do this it is sufficient to construct a single counter-example, so consider a 
mixture of two components in which one of those components is itself a mixture of 
two components, with mixing coefficients given by linear-logistic models. Show that 
this cannot be represented by a single-level mixture of 3 components having mixing 
coefficients determined by a linear-softmax model. 


Appendix A. Data Sets 


In this appendix, we give a brief introduction to the data sets used to illustrate some 
of the algorithms described in this book. Detailed information on file formats for 
these data sets, as well as the data files themselves, can be obtained from the book 
web site: 


http://research.microsoft.com/~cmbishop/PRML 


Handwritten Digits 


The digits data used in this book is taken from the MNIST data set (LeCun et al., 
1998), which itself was constructed by modifying a subset of the much larger data 
set produced by NIST (the National Institute of Standards and Technology). It com- 
prises a training set of 60,000 examples and a test set of 10,000 examples. Some 
of the data was collected from Census Bureau employees and the rest was collected 
from high-school children, and care was taken to ensure that the test examples were 
written by different individuals to the training examples. 

The original NIST data had binary (black or white) pixels. To create MNIST, 
these images were size normalized to fit in a 20 x 20 pixel box while preserving their 
aspect ratio. As a consequence of the anti-aliasing used to change the resolution of 
the images, the resulting MNIST digits are grey scale. These images were then 
centred in a 28 x 28 box. Examples of the MNIST digits are shown in Figure A.1. 

Error rates for classifying the digits range from 12% for a simple linear classi- 
fier, through 0.56% for a carefully designed support vector machine, to 0.4% for a 
convolutional neural network (LeCun et al., 1998). 
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Figure A.1 


One hundred examples of the 
MNIST digits chosen at ran- 
dom from the training set. 
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Oil Flow 


This is a synthetic data set that arose out of a project aimed at measuring nonin- 
vasively the proportions of oil, water, and gas in North Sea oil transfer pipelines 
(Bishop and James, 1993). It is based on the principle of dual-energy gamma densit- 
ometry. The ideas is that if a narrow beam of gamma rays is passed through the pipe, 
the attenuation in the intensity of the beam provides information about the density of 
material along its path. Thus, for instance, the beam will be attenuated more strongly 
by oil than by gas. 

A single attenuation measurement alone is not sufficient because there are two 
degrees of freedom corresponding to the fraction of oil and the fraction of water (the 
fraction of gas is redundant because the three fractions must add to one). To address 
this, two gamma beams of different energies (in other words different frequencies or 
wavelengths) are passed through the pipe along the same path, and the attenuation of 
each is measured. Because the absorbtion properties of different materials vary dif- 
ferently as a function of energy, measurement of the attenuations at the two energies 
provides two independent pieces of information. Given the known absorbtion prop- 
erties of oil, water, and gas at the two energies, it is then a simple matter to calculate 
the average fractions of oil and water (and hence of gas) measured along the path of 
the gamma beams. 

There is a further complication, however, associated with the motion of the ma- 
terials along the pipe. If the flow velocity is small, then the oil floats on top of the 
water with the gas sitting above the oil. This is known as a laminar or stratified 
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The three geometrical configurations of the oil, 
water, and gas phases used to generate the oil- 
flow data set. For each configuration, the pro- 
portions of the three phases can vary. 


Stratified Annular 


Boil 
BB Water 


Gas 


E Mix 


Homogeneous 


flow configuration and is illustrated in Figure A.2. As the flow velocity is increased, 
more complex geometrical configurations of the oil, water, and gas can arise. For the 
purposes of this data set, two specific idealizations are considered. In the annular 
configuration the oil, water, and gas form concentric cylinders with the water around 
the outside and the gas in the centre, whereas in the homogeneous configuration the 
oil, water and gas are assumed to be intimately mixed as might occur at high flow 
velocities under turbulent conditions. These configurations are also illustrated in 
Figure A.2. 

We have seen that a single dual-energy beam gives the oil and water fractions 
measured along the path length, whereas we are interested in the volume fractions of 
oil and water. This can be addressed by using multiple dual-energy gamma densit- 
ometers whose beams pass through different regions of the pipe. For this particular 
data set, there are six such beams, and their spatial arrangement is shown in Fig- 
ure A.3. A single observation is therefore represented by a 12-dimensional vector 
comprising the fractions of oil and water measured along the paths of each of the 
beams. We are, however, interested in obtaining the overall volume fractions of the 
three phases in the pipe. This is much like the classical problem of tomographic re- 
construction, used in medical imaging for example, in which a two-dimensional dis- 


Cross section of the pipe showing the arrangement of the 
six beam lines, each of which comprises a single dual- 
energy gamma densitometer. Note that the vertical beams 
are asymmetrically arranged relative to the central axis 
(shown by the dotted line). 
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tribution is to be reconstructed from an number of one-dimensional averages. Here 
there are far fewer line measurements than in a typical tomography application. On 
the other hand the range of geometrical configurations is much more limited, and so 
the configuration, as well as the phase fractions, can be predicted with reasonable 
accuracy from the densitometer data. 

For safety reasons, the intensity of the gamma beams is kept relatively weak and 
so to obtain an accurate measurement of the attenuation, the measured beam intensity 
is integrated over a specific time interval. For a finite integration time, there are 
random fluctuations in the measured intensity due to the fact that the gamma beams 
comprise discrete packets of energy called photons. In practice, the integration time 
is chosen as a compromise between reducing the noise level (which requires a long 
integration time) and detecting temporal variations in the flow (which requires a short 
integration time). The oil flow data set is generated using realistic known values for 
the absorption properties of oil, water, and gas at the two gamma energies used, and 
with a specific choice of integration time (10 seconds) chosen as characteristic of a 
typical practical setup. 

Each point in the data set is generated independently using the following steps: 


1. Choose one of the three phase configurations at random with equal probability. 


2. Choose three random numbers f1, f2 and fz from the uniform distribution over 
(0, 1) and define 


O oe 
fit fot fs’ oe fit fot fs 


This treats the three phases on an equal footing and ensures that the volume 
fractions add to one. 


foi = (A.1) 


3. For each of the six beam lines, calculate the effective path lengths through oil 
and water for the given phase configuration. 


4. Perturb the path lengths using the Poisson distribution based on the known 
beam intensities and integration time to allow for the effect of photon statistics. 


Each point in the data set comprises the 12 path length measurements, together 
with the fractions of oil and water and a binary label describing the phase configu- 
ration. The data set is divided into training, validation, and test sets, each of which 
comprises 1,000 independent data points. Details of the data format are available 
from the book web site. 

In Bishop and James (1993), statistical machine learning techniques were used 
to predict the volume fractions and also the geometrical configuration of the phases 
shown in Figure A.2, from the 12-dimensional vector of measurements. The 12- 
dimensional observation vectors can also be used to test data visualization algo- 
rithms. 

This data set has a rich and interesting structure, as follows. For any given 
configuration there are two degrees of freedom corresponding to the fractions of 


Figure A.4 
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oil and water, and so for infinite integration time the data will locally live on a two- 
dimensional manifold. For a finite integration time, the individual data points will be 
perturbed away from the manifold by the photon noise. In the homogeneous phase 
configuration, the path lengths in oil and water are linearly related to the fractions of 
oil and water, and so the data points lie close to a linear manifold. For the annular 
configuration, the relationship between phase fraction and path length is nonlinear 
and so the manifold will be nonlinear. In the case of the laminar configuration the 
situation is even more complex because small variations in the phase fractions can 
cause one of the horizontal phase boundaries to move across one of the horizontal 
beam lines leading to a discontinuous jump in the 12-dimensional observation space. 
In this way, the two-dimensional nonlinear manifold for the laminar configuration is 
broken into six distinct segments. Note also that some of the manifolds for different 
phase configurations meet at specific points, for example if the pipe is filled entirely 
with oil, it corresponds to specific instances of the laminar, annular, and homoge- 
neous configurations. 


Old Faithful 


Old Faithful, shown in Figure A.4, is a hydrothermal geyser in Yellowstone National 
Park in the state of Wyoming, U.S.A., and is a popular tourist attraction. Its name 
stems from the supposed regularity of its eruptions. 

The data set comprises 272 observations, each of which represents a single erup- 
tion and contains two variables corresponding to the duration in minutes of the erup- 
tion, and the time until the next eruption, also in minutes. Figure A.5 shows a plot of 
the time to the next eruption versus the duration of the eruptions. It can be seen that 
the time to the next eruption varies considerably, although knowledge of the duration 
of the current eruption allows it to be predicted more accurately. Note that there exist 
several other data sets relating to the eruptions of Old Faithful. 


The Old Faithful geyser 
in Yellowstone National 
Park. ©Bruce T. Gourley 
www.brucegourley.com. 
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Figure A.5 Plot of the time to the next eruption 
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in minutes (vertical axis) versus the 100 


duration of the eruption in minutes 
(horizontal axis) for the Old Faithful 90 | 
data set. 
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Synthetic Data 


Throughout the book, we use two simple synthetic data sets to illustrate many of the 
algorithms. The first of these is a regression problem, based on the sinusoidal func- 
tion, shown in Figure A.6. The input values {x„ } are generated uniformly in range 
(0,1), and the corresponding target values {t,,} are obtained by first computing the 
corresponding values of the function sin(272), and then adding random noise with 
a Gaussian distribution having standard deviation 0.3. Various forms of this data set, 
having different numbers of data points, are used in the book. 

The second data set is a classification problem having two classes, with equal 
prior probabilities, and is shown in Figure A.7. The blue class is generated from a 
single Gaussian while the red class comes from a mixture of two Gaussians. Be- 
cause we know the class priors and the class-conditional densities, it is straightfor- 
ward to evaluate and plot the true posterior probabilities as well as the minimum 
misclassification-rate decision boundary, as shown in Figure A.7. 
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© 


0 x 1 0 x 1 
Figure A.6 The left-hand plot shows the synthetic regression data set along with the underlying sinusoidal 
function from which the data points were generated. The right-hand plot shows the true conditional distribution 


p(t|x) from which the labels are generated, in which the green curve denotes the mean, and the shaded region 
spans one standard deviation on each side of the mean. 


—2 + 


-2 0 2 


-2 
Figure A.7 The left plot shows the synthetic classification data set with data from the two classes shown in 
red and blue. On the right is a plot of the true posterior probabilities, shown on a colour scale going from pure 
red denoting probability of the red class is 1 to pure blue denoting probability of the red class is 0. Because 
these probabilities are known, the optimal decision boundary for minimizing the misclassification rate (which 
corresponds to the contour along which the posterior probabilities for each class equal 0.5) can be evaluated 
and is shown by the green curve. This decision boundary is also plotted on the left-hand figure. 


Appendix B. Probability Distributions 


In this appendix, we summarize the main properties of some of the most widely used 
probability distributions, and for each distribution we list some key statistics such as 


the expectation 


E[x], the variance (or covariance), the mode, and the entropy H[x]. 


All of these distributions are members of the exponential family and are widely used 
as building blocks for more sophisticated probabilistic models. 


Bernoulli 


This is the distribution for a single binary variable x € {0,1} representing, for 
example, the result of flipping a coin. It is governed by a single continuous parameter 
u € [0, 1] that represents the probability of x = 1. 


Bern(z|u) = w*(1—p)'* (B.1) 
Elz] = p (B.2) 
var[z] = a(l- u) (B.3) 

= f 1 ifp>os, 
me = { 0 otherwise ee 


H[z] = —plnp—(1—-p)In(1 — p). (B.5) 


The Bernoulli is a special case of the binomial distribution for the case of a single 
observation. Its conjugate prior for ju is the beta distribution. 
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Beta 


This is a distribution over a continuous variable u € [0,1], which is often used to 
represent the probability for some binary event. It is governed by two parameters a 
and b that are constrained by a > 0 and b > 0 to ensure that the distribution can be 
normalized. 


T(a+b 
Beta(yula,b) = Tame = p (B.6) 
El = 95 mag 
ab 
Ven) <> a ee) 
= 
modeļu] = ener (B.9) 


The beta is the conjugate prior for the Bernoulli distribution, for which a and b can 
be interpreted as the effective prior number of observations of x = 1 and x = 0, 
respectively. Its density is finite if a > 1 and b > 1, otherwise there is a singularity 
at u = 0 and/or u = 1. For a = b = 1, it reduces to a uniform distribution. The beta 
distribution is a special case of the K-state Dirichlet distribution for K = 2. 


Binomial 


The binomial distribution gives the probability of observing m occurrences of x = 1 
in a set of N samples from a Bernoulli distribution, where the probability of observ- 
ing x = lis u € [0,1]. 


Bin(m|N, u) = a FT sil ES (B.10) 
[m] = Np (B.11) 

var[m] = Ny(1—p) (B.12) 
mode[m] = |(N+1)p| (B.13) 


where | (V + 1)| denotes the largest integer that is less than or equal to (N + 1), 


and the quantity 
Ae ab (B.14) 
mj}  m!(N—m)! f 


denotes the number of ways of choosing m objects out of a total of N identical 
objects. Here m!, pronounced ‘factorial m’, denotes the product m x (m — 1) x 
..., X2 x 1. The particular case of the binomial distribution for N = 1 is known as 
the Bernoulli distribution, and for large N the binomial distribution is approximately 
Gaussian. The conjugate prior for p is the beta distribution. 
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Dirichlet 


The Dirichlet is a multivariate distribution over K random variables 0 < uk <S 1, 


where k = 1,..., K, subject to the constraints 
K 
O< pe <1, 5 ei (B.15) 
k=1 
Denoting u = (41,-.., uK)" and @ = (a1, ..., aK)", we have 
K 
Dir(ula) = Cla) J |p (B.16) 
k=1 
Bju] = Æ (B.17) 
Q 
aR (a T ag) 
— Wea- r) B.1 
var [WK (a+ 1) (B.18) 
; = QjQk 
cov[ujur] = PCES i) (B.19) 
_ Ak — 1 
mode|ux] = ZK (B.20) 
Ejln uk] = w(ax) — v(@) (B.21) 
K 
Hju] = -X (ax -1) {v(ax)-v(@}-mC(a) B22) 
k=1 
where a 
a 
Cla) = > B.23 
(©) = Ten)--- Tax) = 
and 
K 
a= $. Qk. (B.24) 
k=1 
Here d 
pla) = u” T(a) (B.25) 


is known as the digamma function (Abramowitz and Stegun, 1965). The parameters 
Qz are subject to the constraint a; > 0 in order to ensure that the distribution can be 
normalized. 

The Dirichlet forms the conjugate prior for the multinomial distribution and rep- 
resents a generalization of the beta distribution. In this case, the parameters œg can 
be interpreted as effective numbers of observations of the corresponding values of 
the k-dimensional binary observation vector x. As with the beta distribution, the 
Dirichlet has finite density everywhere provided a; > 1 for all k. 
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Gamma 


The Gamma is a probability distribution over a positive random variable r > 0 
governed by parameters a and b that are subject to the constraints a > 0 and b > 0 
to ensure that the distribution can be normalized. 


— 1 a_a—1_—br 

Gam(rtla,b) = Tay’ Toe (B.26) 

a 
a = 2 B.27 
T 5 (B.27) 

a 
varir] = Rp (B.28) 

a— 1 

modeļr| = 5 fr a>1 (B.29) 
Ellnr] = w(a)—Inb (B.30) 
H[7] = InI(a) —(a—1)¥(a) —-Inb+a (B.31) 


where y(-) is the digamma function defined by (B.25). The gamma distribution is 
the conjugate prior for the precision (inverse variance) of a univariate Gaussian. For 
a > 1 the density is everywhere finite, and the special case of a = 1 is known as the 
exponential distribution. 


Gaussian 


The Gaussian is the most widely used distribution for continuous variables. It is also 
known as the normal distribution. In the case of a single variable x € (—o00, 00) it is 
governed by two parameters, the mean u € (—oo, 00) and the variance o? > 0. 


1 1 
Niele") = een? 53 (# we} (B.32) 
Elz] = u (B.33) 
var[z] = o° (B.34) 
mode|[z] = p (B.35) 
H[z] = sino? +5 (1+In(2n)). (B.36) 


The inverse of the variance r = 1/o° is called the precision, and the square root 
of the variance ø is called the standard deviation. The conjugate prior for ju is the 
Gaussian, and the conjugate prior for 7 is the gamma distribution. If both 4 and T 
are unknown, their joint conjugate prior is the Gaussian-gamma distribution. 

For a D-dimensional vector x, the Gaussian is governed by a D-dimensional 
mean vector ys and a D x D covariance matrix 4 that must be symmetric and 
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positive-definite. 


N(x|u, £) = a IEE exp { (x p)7 a(x — n} (B.37) 
ajx] = p (B.38) 
cov[x] = 5 (B.39) 
mode[x] = p (B.40) 

1 D 
H[x| = 5 In || + 5 (1+ In(27)). (B.41) 


The inverse of the covariance matrix A = © ' is the precision matrix, which is also 
symmetric and positive definite. Averages of random variables tend to a Gaussian, by 
the central limit theorem, and the sum of two Gaussian variables is again Gaussian. 
The Gaussian is the distribution that maximizes the entropy for a given variance 
(or covariance). Any linear transformation of a Gaussian random variable is again 
Gaussian. The marginal distribution of a multivariate Gaussian with respect to a 
subset of the variables is itself Gaussian, and similarly the conditional distribution is 
also Gaussian. The conjugate prior for yz is the Gaussian, the conjugate prior for A 
is the Wishart, and the conjugate prior for (u, A) is the Gaussian-Wishart. 

If we have a marginal Gaussian distribution for x and a conditional Gaussian 
distribution for y given x in the form 


p(x) N (x|u, A~*) (B.42) 
plylx) = N(y|Ax+b,L~*) (B.43) 


then the marginal distribution of y, and the conditional distribution of x given y, are 
given by 


py) = N(y|Auw+b,L7'+ AAA") (B.44) 
p(xly) = N(x|={A*L(y — b) + Ap}, >) (B.45) 

where 
S=(A+ ATLA)". (B.46) 


—1 


If we have a joint Gaussian distribution N (x|u, =) with A = X~- and we 


define the following partitions 


_ [Xa — [Ha 
<= (E) w= (1) Bar 


Yaa Dab Noa Nab 
X = A= B.48 
& 3 i e | (PAS) 
then the conditional distribution p(xa|x») is given by 
P(Xalx0) = N(x|HMajn» Aca) (B.49) 


Hajo = Ba — Aga Nab(Xp — Mp) (B.50) 
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and the marginal distribution p(x,) is given by 


P(Xa) = N (Xe | Bas Xaa): (B.51) 


Gaussian-Gamma 


This is the conjugate prior distribution for a univariate Gaussian N (x|u, A7!) in 
which the mean y and the precision À are both unknown and is also called the 
normal-gamma distribution. It comprises the product of a Gaussian distribution for 
u, whose precision is proportional to À, and a gamma distribution over À. 


plu, Alo, 8, a,b) =N (uļuo, (8A) *) Gam(AJa, b). (B.52) 


Gaussian-Wishart 


This is the conjugate prior distribution for a multivariate Gaussian N (x|u, A) in 
which both the mean yz and the precision A are unknown, and is also called the 
normal- Wishart distribution. It comprises the product of a Gaussian distribution for 
H, whose precision is proportional to A, and a Wishart distribution over A. 


plu, Alto, 8,W,v) =N (ujma (GA)~*) W(A|W, v). (B.53) 


For the particular case of a scalar x, this is equivalent to the Gaussian-gamma distri- 
bution. 


Multinomial 


If we generalize the Bernoulli distribution to an K-dimensional binary variable x 
with components x, € {0,1} such that 5 p Zk = 1, then we obtain the following 
discrete distribution 


K 
p(x) = [ee (B.54) 
k=1 
Eje] = ur (B.55) 
var|zy~] = px(1 — uk) (B.56) 
cov[zjzk] = Juin (B.57) 


M 
H[x]} = — D uk lN Uk (B.58) 
k=1 
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where J;;, is the j, k element of the identity matrix. Because p(x, = 1) = pg, the 
parameters must satisfy 0 < up < 1 and yr ba = 1. 

The multinomial distribution is a multivariate generalization of the binomial and 
gives the distribution over counts mę for a K-state discrete variable to be in state k 
given a total number of observations N. 


M 
N 
Mult(m,,m2,...,MK|u,N) = ( ) [Le (B.59) 
mMımM2... MM kel 
var[mx| = Nuz(1— pr) (B.61) 
cov[mjmk] = —Nujuk (B.62) 


where ps = (1, ..-, ug)”, and the quantity 


(nim, m) = Sl B63) 


MiM... MK mı!... mg! 


gives the number of ways of taking N identical objects and assigning m,, of them to 
bin k fork = 1,..., K. The value of up gives the probability of the random variable 
taking state k, and so these parameters are subject to the constraints 0 < pw, < 1 
and $`, uk = 1. The conjugate prior distribution for the parameters {pux} is the 
Dirichlet. 


Normal 


The normal distribution is simply another name for the Gaussian. In this book, we 
use the term Gaussian throughout, although we retain the conventional use of the 
symbol M to denote this distribution. For consistency, we shall refer to the normal- 
gamma distribution as the Gaussian-gamma distribution, and similarly the normal- 
Wishart is called the Gaussian- Wishart. 


Student’s t 


This distribution was published by William Gosset in 1908, but his employer, Gui- 
ness Breweries, required him to publish under a pseudonym, so he chose ‘Student’. 
In the univariate form, Student’s t-distribution is obtained by placing a conjugate 
gamma prior over the precision of a univariate Gaussian distribution and then inte- 
grating out the precision variable. It can therefore be viewed as an infinite mixture 
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of Gaussians having the same mean but different variances. 


1/2 97 —v/2-1/2 
St(x|u, À, v) a a ( A ) h + Me] (B.64) 


T(v/2) TV V 
Elze] = u for y>1 (B.65) 
1 v 
var[z] = v? for v > 2 (B.66) 
mode[z] = p. (B.67) 


Here v > 0 is called the number of degrees of freedom of the distribution. The 
particular case of v = 1 is called the Cauchy distribution. 

For a D-dimensional variable x, Student’s t-distribution corresponds to marginal- 
izing the precision matrix of a multivariate Gaussian with respect to a conjugate 
Wishart prior and takes the form 


St(x|u, A, v) r wi + L 2) ae | | =| —— (B.68) 
Elx] = pe for v>1 (B.69) 
covik] = = a JA forv>2 (B.70) 
mode[x] = p (B.71) 
where A? is the squared Mahalanobis distance defined by 
A? = (x — p)7 A(x — p). (B.72) 


In the limit v — oo, the t-distribution reduces to a Gaussian with mean pu and pre- 
cision A. Student’s t-distribution provides a generalization of the Gaussian whose 
maximum likelihood parameter values are robust to outliers. 


Uniform 


This is a simple distribution for a continuous variable x defined over a finite interval 
x € [a,b] where b > a. 


U(z|a,b) = ae (B.73) 
tig a 5 z (B.74) 

_ (b-a? 
var[z] = a (B.75) 
H[z] = In(b- a). (B.76) 


If x has distribution U(2|0, 1), then a + (b — a)x will have distribution U(z|a, b). 
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Von Mises 
The von Mises distribution, also known as the circular normal or the circular Gaus- 
sian, is a univariate Gaussian-like periodic distribution for a variable 0 € [0, 27). 


p(9|00,m) exp {mcos(@ — 0) } (B.77) 


1 
27Io(m) 
where To(m) is the zeroth-order Bessel function of the first kind. The distribution 
has period 27 so that p(@ + 27) = p(@) for all 6. Care must be taken in interpret- 
ing this distribution because simple expectations will be dependent on the (arbitrary) 
choice of origin for the variable 6. The parameter 6, is analogous to the mean of a 
univariate Gaussian, and the parameter m > 0, known as the concentration param- 
eter, is analogous to the precision (inverse variance). For large m, the von Mises 
distribution is approximately a Gaussian centred on ĝo. 


Wishart 


The Wishart distribution is the conjugate prior for the precision matrix of a multi- 
variate Gaussian. 


W(A|W, v) = B(W, v) |A| P79? exp (-;™w-a)) (B.78) 
where 
D i=; =a 
B(W,v) = lw |’? (27 geome TF (==) (B.79) 
i=l 
EJA] = vw (B.80) 
= v+1—i 
E[In|A\] = yo a ) + DIn2+ln|W] (B.81) 
i=1 
H[A] = -mB(W, y) - EPDE mja 2 (B.82) 


where W is a D x D symmetric, positive definite matrix, and 7)(-) is the digamma 
function defined by (B.25). The parameter v is called the number of degrees of 
freedom of the distribution and is restricted to v > D — 1 to ensure that the Gamma 
function in the normalization factor is well-defined. In one dimension, the Wishart 
reduces to the gamma distribution Gam(Ala, b) given by (B.26) with parameters 
a = v/2 and b = 1/2W. 


Appendix C. Properties of Matrices 


In this appendix, we gather together some useful properties and identities involving 
matrices and determinants. This is not intended to be an introductory tutorial, and 
it is assumed that the reader is already familiar with basic linear algebra. For some 
results, we indicate how to prove them, whereas in more complex cases we leave 
the interested reader to refer to standard textbooks on the subject. In all cases, we 
assume that inverses exist and that matrix dimensions are such that the formulae 
are correctly defined. A comprehensive discussion of linear algebra can be found in 
Golub and Van Loan (1996), and an extensive collection of matrix properties is given 
by Lutkepohl (1996). Matrix derivatives are discussed in Magnus and Neudecker 
(1999). 


Basic Matrix Identities 


A matrix A has elements A;; where 7 indexes the rows, and 7 indexes the columns. 
We use Iy to denote the N x N identity matrix (also called the unit matrix), and 
where there is no ambiguity over dimensionality we simply use I. The transpose 
matrix AT has elements (AT); = Aji. From the definition of transpose, we have 


(AB)'=B'A™ (C.1) 


which can be verified by writing out the indices. The inverse of A, denoted A~}, 
satisfies 
AA! = AA =I. (C.2) 


Because ABB~!A~—! =I, we have 
(AB)! = BA. (C.3) 
Also we have 


(AT) =(A7)" (C4) 
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which is easily proven by taking the transpose of (C.2) and applying (C.1). 
A useful identity involving matrix inverses is the following 


(P-'+B'R'B)'B'R' = PB’"(BPB’ + R)*?. (C.5) 


which is easily verified by right multiplying both sides by (BPB* + R). Suppose 
that P has dimensionality N x N while R has dimensionality M x M, so that B is 
M x N. Thenif M < N, it will be much cheaper to evaluate the right-hand side of 
(C.5) than the left-hand side. A special case that sometimes arises is 


(I+ AB)~'A = A(I+BA)7}. (C.6) 


Another useful identity involving inverses is the following: 


(A +BD'C)™! = A~ — A“! B(D+ CA™'B)"'CA7! (C7) 


which is known as the Woodbury identity and which can be verified by multiplying 
both sides by (A + BD~!C). This is useful, for instance, when A is large and 
diagonal, and hence easy to invert, while B has many rows but few columns (and 
conversely for C) so that the right-hand side is much cheaper to evaluate than the 
left-hand side. 

A set of vectors {a,,..., ay} is said to be linearly independent if the relation 
ae Qnan = 0 holds only if all ap = 0. This implies that none of the vectors 
can be expressed as a linear combination of the remainder. The rank of a matrix is 
the maximum number of linearly independent rows (or equivalently the maximum 
number of linearly independent columns). 


Traces and Determinants 


Trace and determinant apply to square matrices. The trace Tr(A) of a matrix A 
is defined as the sum of the elements on the leading diagonal. By writing out the 
indices, we see that 

Tr(AB) = Tr(BA). (C.8) 


By applying this formula multiple times to the product of three matrices, we see that 
Tr(ABC) = Tr(CAB) = Tr(BCA) (C.9) 


which is known as the cyclic property of the trace operator and which clearly extends 
to the product of any number of matrices. The determinant |A| of an N x N matrix 
A is defined by 


JA] = So (41) Asi, Aoi, +++ Ani (C.10) 


in which the sum is taken over all products consisting of precisely one element from 
each row and one element from each column, with a coefficient +1 or —1 according 
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to whether the permutation 7,72... iy is even or odd, respectively. Note that |I| = 1. 
Thus, for a 2 x 2 matrix, the determinant takes the form 


Q11 G12 
a21 422 


|A| = = 411422 — 412491. (C.11) 


The determinant of a product of two matrices is given by 
|AB| = |AI|B| C.12) 


as can be shown from (C.10). Also, the determinant of an inverse matrix is given by 


1 
A |= —_ (C.13) 
JAM =a 
which can be shown by taking the determinant of (C.2) and applying (C.12). 
If A and B are matrices of size N x M, then 
[Iy + AB"| = [In + AB]. (C.14) 
A useful special case is 
|Iy +ab™| = 1 + afb (C.15) 


where a and b are N-dimensional column vectors. 


Matrix Derivatives 


Sometimes we need to consider derivatives of vectors and matrices with respect to 
scalars. The derivative of a vector a with respect to a scalar z is itself a vector whose 


components are given by 
ða ð Qi 
— j] = C.16 
( Ox ) , oO om 


with an analogous definition for the derivative of a matrix. Derivatives with respect 
to vectors and matrices can also be defined, for instance 


Ox Ox 

6) oor (C.17) 
and similarly 
ða Oa; 
Sy) eee l 
(2) Ob; ee 
ij 
The following is easily proven by writing out the components 
o o 

z (xta) = a (atx) =a. (C.19) 
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Similarly 
o OA OB 
AB) = B+A—. 2 

Ox ee) Ox Ox ee 
The derivative of the inverse of a matrix can be expressed as 

o OA 

— (At) =-A7'——A™ C.21 

Ox ( ) Ox € ) 


as can be shown by differentiating the equation A~'A = I using (C.20) and then 
right multiplying by A~'. Also 


a ðA 


which we shall prove later. If we choose zx to be one of the elements of A, we have 


o 


as can be seen by writing out the matrices using index notation. We can write this 
result more compactly in the form 


ð pT 
za T (AB) = B". (C.24) 


With this notation, we have the following properties 


O T E 

5a Tt (A B) = B (C.25) 
o 

ð T = T 


which can again be proven by writing out the matrix indices. We also have 


o Femi? 
za IAI = (A‘) (C.28) 


which follows from (C.22) and (C.26). 


Eigenvector Equation 
For a square matrix A of size M x M, the eigenvector equation is defined by 


Au; = Aiu; (C.29) 
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fori = 1,..., M, where u; is an eigenvector and A; is the corresponding eigenvalue. 
This can be viewed as a set of M simultaneous homogeneous linear equations, and 
the condition for a solution is that 


|A — Al] =0 (C.30) 


which is known as the characteristic equation. Because this is a polynomial of order 
M in Xj, it must have M solutions (though these need not all be distinct). The rank 
of A is equal to the number of nonzero eigenvalues. 

Of particular interest are symmetric matrices, which arise as covariance ma- 
trices, kernel matrices, and Hessians. Symmetric matrices have the property that 
Aij = Aji, or equivalently AT = A. The inverse of a symmetric matrix is also sym- 
metric, as can be seen by taking the transpose of A~'A = I and using AA~' = I 
together with the symmetry of I. 

In general, the eigenvalues of a matrix are complex numbers, but for symmetric 
matrices the eigenvalues \; are real. This can be seen by first left multiplying (C.29) 
by (u*)™, where x denotes the complex conjugate, to give 


(uf)" Au; = A; (uf)" wi. (C31) 
Next we take the complex conjugate of (C.29) and left multiply by uf to give 
uj Au% = peur už. (C.32) 


where we have used A* = A because we consider only real matrices A. Taking 
the transpose of the second of these equations, and using AT = A, we see that the 
left-hand sides of the two equations are equal, and hence that Až = A; and so A; 
must be real. 

The eigenvectors u; of a real symmetric matrix can be chosen to be orthonormal 
(1.e., orthogonal and of unit length) so that 


uj uj = Tiz (C.33) 


where J; are the elements of the identity matrix I. To show this, we first left multiply 
(C.29) by u7 to give 
uj; Au; E Au; u; (C.34) 


and hence, by exchange of indices, we have 
u; Au; = Aju; uj. (0.35) 


We now take the transpose of the second equation and make use of the symmetry 
property AT = A, and then subtract the two equations to give 


(A; — Aj) uj u; = 0. (C.36) 


Hence, for A; 4 Aj, we have u; uj = 0, and hence u; and u; are orthogonal. If the 


two eigenvalues are equal, then any linear combination au; + ĝu; is also an eigen- 
vector with the same eigenvalue, so we can select one linear combination arbitrarily, 
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and then choose the second to be orthogonal to the first (it can be shown that the de- 
generate eigenvectors are never linearly dependent). Hence the eigenvectors can be 
chosen to be orthogonal, and by normalizing can be set to unit length. Because there 
are M eigenvalues, the corresponding M orthogonal eigenvectors form a complete 
set and so any (/-dimensional vector can be expressed as a linear combination of 
the eigenvectors. 

We can take the eigenvectors u; to be the columns of an M x M matrix U, 
which from orthonormality satisfies 


UTU=L. (C.37) 


Such a matrix is said to be orthogonal. Interestingly, the rows of this matrix are also 

orthogonal, so that UUT = I. To show this, note that (C.37) implies U'TUU~! = 

U`! = UT and so UU! = UUT = I. Using (C.12), it also follows that |U| = 1. 
The eigenvector equation (C.29) can be expressed in terms of U in the form 


AU=UA (C.38) 


where A is an M x M diagonal matrix whose diagonal elements are given by the 
eigenvalues \;. 
If we consider a column vector x that is transformed by an orthogonal matrix U 
to give a new vector 
x=Ux (C.39) 


then the length of the vector is preserved because 

XTX = x"UTUx = x"x (C.40) 
and similarly the angle between any two such vectors is preserved because 

xTy = x "UTUy = x"y. (C.41) 


Thus, multiplication by U can be interpreted as a rigid rotation of the coordinate 
system. 
From (C.38), it follows that 


UTAU =A (C.42) 


and because A is a diagonal matrix, we say that the matrix A is diagonalized by the 
matrix U. If we left multiply by U and right multiply by UT, we obtain 


A = UAUT (C.43) 


Taking the inverse of this equation, and using (C.3) together with U-! = UT, we 
have 
A+ = UA™UT. (C.44) 
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These last two equations can also be written in the form 
A = X Muu] (C.45) 
At = So uu}. (C.46) 
If we take the determinant of (C.43), and use (C.12), we obtain 
M 
|Al =] [> (C.47) 
i=1 


Similarly, taking the trace of (C.43), and using the cyclic property (C.8) of the trace 
operator together with UTU = I, we have 


M 
Tr(A) = XDA: (C.48) 
i=1 


We leave it as an exercise for the reader to verify (C.22) by making use of the results 
(C.33), (C.45), (C.46), and (C.47). 

A matrix A is said to be positive definite, denoted by A > 0, if wT Aw > 0 for 
all values of the vector w. Equivalently, a positive definite matrix has A; > 0 for all 
of its eigenvalues (as can be seen by setting w to each of the eigenvectors in turn, 
and by noting that an arbitrary vector can be expanded as a linear combination of the 
eigenvectors). Note that positive definite is not the same as all the elements being 


positive. For example, the matrix 
1 2 
( 3 4 ) (C.49) 


has eigenvalues A, ~ 5.37 and ào ~ —0.37. A matrix is said to be positive semidef- 
inite if w' Aw > 0 holds for all values of w, which is denoted A > 0, and is 
equivalent to A; > 0. 


Appendix D. Calculus of Variations 


We can think of a function y(x) as being an operator that, for any input value z, 
returns an output value y. In the same way, we can define a functional Fy] to be 
an operator that takes a function y(x) and returns an output value F. An example of 
a functional is the length of a curve drawn in a two-dimensional plane in which the 
path of the curve is defined in terms of a function. In the context of machine learning, 
a widely used functional is the entropy H|z] for a continuous variable z because, for 
any choice of probability density function p(x), it returns a scalar value representing 
the entropy of x under that density. Thus the entropy of p(x) could equally well have 
been written as H[p]. 

A common problem in conventional calculus is to find a value of x that max- 
imizes (or minimizes) a function y(x). Similarly, in the calculus of variations we 
seek a function y(x) that maximizes (or minimizes) a functional F'[y]. That is, of all 
possible functions y(x), we wish to find the particular function for which the func- 
tional F'[y] is a maximum (or minimum). The calculus of variations can be used, for 
instance, to show that the shortest path between two points is a straight line or that 
the maximum entropy distribution is a Gaussian. 

If we weren’t familiar with the rules of ordinary calculus, we could evaluate a 
conventional derivative dy/dx by making a small change « to the variable x and 
then expanding in powers of e, so that 


d 
ylz +€) = y(x) + Te + O(e) (D.1) 
and finally taking the limit e — 0. Similarly, for a function of several variables 
y(%1,...,@p), the corresponding partial derivatives are defined by 
Ld 
y(z1 + €1,-..,@p + €p) = Y(z1,-.-,£p) + 3 ee +0(2). D2 


The analogous definition of a functional derivative arises when we consider how 
much a functional F'[y] changes when we make a small change en(x) to the function 
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Figure D.1 A functional derivative can be defined by 
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considering how the value of a functional 
Fy] changes when the function y(x) is 
changed to y(x) + en(x) where n(x) is an 
arbitrary function of x. 


y(x), where n(x) is an arbitrary function of x, as illustrated in Figure D.1. We denote 
the functional derivative of E[f] with respect to f(x) by 6F'/d f(x), and define it by 
the following relation: 


OF 
Fiyle) + ena) = Fe) +e | Egede D3) 
This can be seen as a natural extension of (D.2) in which F'[y] now depends on a 
continuous set of variables, namely the values of y at all points x. Requiring that the 
functional be stationary with respect to small variations in the function y(x) gives 


f e) dz = 0. (D.4) 


Because this must hold for an arbitrary choice of n(x), it follows that the functional 
derivative must vanish. To see this, imagine choosing a perturbation n(x) that is zero 
everywhere except in the neighbourhood of a point 7, in which case the functional 
derivative must be zero at x = 7. However, because this must be true for every 
choice of T, the functional derivative must vanish for all values of z. 

Consider a functional that is defined by an integral over a function G(y, y’, x) 
that depends on both y(x) and its derivative y'(x) as well as having a direct depen- 
dence on x 


Fly] = f COM aja (D5) 


where the value of y(x) is assumed to be fixed at the boundary of the region of 
integration (which might be at infinity). If we now consider variations in the function 
y(x), we obtain 


oG OG 
Fiye) + eno = Fine) +e f SEn) Ea} a+ (2). 6 
We now have to cast this in the form (D.3). To do so, we integrate the second term by 
parts and make use of the fact that n(x) must vanish at the boundary of the integral 
(because y(x) is fixed at the boundary). This gives 


Piya) tenal = Fula +e f AE- E (SE) liaarto ©) 
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from which we can read off the functional derivative by comparison with (D.3). 
Requiring that the functional derivative vanishes then gives 


dG da (aG 
er (55) 0 (D.8) 


which are known as the Euler-Lagrange equations. For example, if 
G = y(2)? +(y'(@))y (D9) 
then the Euler-Lagrange equations take the form 


dy 
y(x) Ja 0. (D.10) 
This second order differential equation can be solved for y(x) by making use of the 
boundary conditions on y(x). 

Often, we consider functionals defined by integrals whose integrands take the 
form G(y, x) and that do not depend on the derivatives of y(x). In this case, station- 
arity simply requires that OG/Oy(x) = 0 for all values of z. 

If we are optimizing a functional with respect to a probability distribution, then 
we need to maintain the normalization constraint on the probabilities. This is often 
most conveniently done using a Lagrange multiplier, which then allows an uncon- 
strained optimization to be performed. 

The extension of the above results to a multidimensional variable x is straight- 
forward. For a more comprehensive discussion of the calculus of variations, see 
Sagan (1969). 


Appendix E. Lagrange Multipliers 


Lagrange multipliers, also sometimes called undetermined multipliers, are used to 
find the stationary points of a function of several variables subject to one or more 
constraints. 

Consider the problem of finding the maximum of a function f (x1, £2) subject to 
a constraint relating xı and 22, which we write in the form 


g(x1,%2) = 0. (E.1) 


One approach would be to solve the constraint equation (E.1) and thus express 22 as 
a function of xı in the form zz = h(a). This can then be substituted into f (21, £2) 
to give a function of x, alone of the form f(x), h(a,)). The maximum with respect 
to xı could then be found by differentiation in the usual way, to give the stationary 
value x}, with the corresponding value of x given by x3 = h(27). 

One problem with this approach is that it may be difficult to find an analytic 
solution of the constraint equation that allows x2 to be expressed as an explicit func- 
tion of xı. Also, this approach treats xı and x2 differently and so spoils the natural 
symmetry between these variables. 

A more elegant, and often simpler, approach is based on the introduction of a 
parameter À called a Lagrange multiplier. We shall motivate this technique from 
a geometrical perspective. Consider a D-dimensional variable x with components 
£1,-..,£p. The constraint equation g(x) = 0 then represents a (DD—1)-dimensional 
surface in x-space as indicated in Figure E.1. 

We first note that at any point on the constraint surface the gradient V g(x) of 
the constraint function will be orthogonal to the surface. To see this, consider a point 
x that lies on the constraint surface, and consider a nearby point x + e that also lies 
on the surface. If we make a Taylor expansion around x, we have 


g(x + €) ~ g(x) + e" V g(x). (E.2) 
Because both x and x + € lie on the constraint surface, we have g(x) = g(x + €) and 


hence e' Vg(x) ~ 0. In the limit ||e|| + 0 we have e"V g(x) = 0, and because e€ is 
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Figure E.1 A geometrical picture of the technique of La- V(x) 

grange multipliers in which we seek to maximize a 

function f(x), subject to the constraint g(x) = 0. 

If x is D dimensional, the constraint g(x) = 0 cor- XA 
responds to a subspace of dimensionality D — 1, 

indicated by the red curve. The problem can 

be solved by optimizing the Lagrangian function 

L(x, A) = f(x) + àg(x). 


g(x) =0 


then parallel to the constraint surface g(x) = 0, we see that the vector Vg is normal 
to the surface. 

Next we seek a point x* on the constraint surface such that f(x) is maximized. 
Such a point must have the property that the vector V f(x) is also orthogonal to the 
constraint surface, as illustrated in Figure E.1, because otherwise we could increase 
the value of f(x) by moving a short distance along the constraint surface. Thus V f 
and Vg are parallel (or anti-parallel) vectors, and so there must exist a parameter A 
such that 

Vf+AVg =0 (E.3) 


where  # 0 is known as a Lagrange multiplier. Note that \ can have either sign. 
At this point, it is convenient to introduce the Lagrangian function defined by 


L(x, A) = f(x) + Ag(x). (E.4) 


The constrained stationarity condition (E.3) is obtained by setting VxL = 0. Fur- 
thermore, the condition 0L/0X = 0 leads to the constraint equation g(x) = 0. 

Thus to find the maximum of a function f(x) subject to the constraint g(x) = 0, 
we define the Lagrangian function given by (E.4) and we then find the stationary 
point of L(x, A) with respect to both x and A. For a D-dimensional vector x, this 
gives D + 1 equations that determine both the stationary point x* and the value of À. 
If we are only interested in x*, then we can eliminate \ from the stationarity equa- 
tions without needing to find its value (hence the term ‘undetermined multiplier’ ). 

As a simple example, suppose we wish to find the stationary point of the function 
f (x1, £2) = 1 — x? — x? subject to the constraint g(x1, 72) = £1 + 2 — 1 = 0, as 
illustrated in Figure E.2. The corresponding Lagrangian function is given by 


L(x, \) = 1-27-23 + A(zı + £2 — 1). (E.5) 


The conditions for this Lagrangian to be stationary with respect to 71, £2, and A give 
the following coupled equations: 


— 2r +A = 0 (E.6) 
—2r> + À (E.T) 


Io 
on 


Figure E.2 


Figure E.3 
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A simple example of the use of Lagrange multipli- 
ers in which the aim is to maximize f(xzı, x2) = 
1 — z? — x3 subject to the constraint g(x1, 22) = 0 
where g(x1, £2) = xı + x2 — 1. The circles show 
contours of the function f (21, x2), and the diagonal 
line shows the constraint surface g(x1, x2) = 0. 


Solution of these equations then gives the stationary point as (xj, 73) = (4, 4), and 
the corresponding value for the Lagrange multiplier is \ = 1. 

So far, we have considered the problem of maximizing a function subject to an 
equality constraint of the form g(x) = 0. We now consider the problem of maxi- 
mizing f(x) subject to an inequality constraint of the form g(x) > 0, as illustrated 
in Figure E.3. 

There are now two kinds of solution possible, according to whether the con- 
strained stationary point lies in the region where g(x) > 0, in which case the con- 
straint is inactive, or whether it lies on the boundary g(x) = 0, in which case the 
constraint is said to be active. In the former case, the function g(x) plays no role 
and so the stationary condition is simply V f(x) = 0. This again corresponds to 
a stationary point of the Lagrange function (E.4) but this time with A = 0. The 
latter case, where the solution lies on the boundary, is analogous to the equality con- 
straint discussed previously and corresponds to a stationary point of the Lagrange 
function (E.4) with A # 0. Now, however, the sign of the Lagrange multiplier is 
crucial, because the function f(x) will only be at a maximum if its gradient is ori- 
ented away from the region g(x) > 0, as illustrated in Figure E.3. We therefore have 
V f(x) = —AV g(x) for some value of À > 0. 

For either of these two cases, the product Ag(x) = 0. Thus the solution to the 


Illustration of the problem of maximizing V f(x) 
f(x) subject to the inequality constraint 
g(x) > 0. 


XA 
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problem of maximizing f(x) subject to g(x) > 0 is obtained by optimizing the 
Lagrange function (E.4) with respect to x and A subject to the conditions 


g(x) > 0 (E.9) 
> 0 (E.10) 
Ag(x) = 0 (E.11) 


These are known as the Karush-Kuhn-Tucker (KKT) conditions (Karush, 1939; Kuhn 
and Tucker, 1951). 

Note that if we wish to minimize (rather than maximize) the function f(x) sub- 
ject to an inequality constraint g(x) > 0, then we minimize the Lagrangian function 
L(x, A) = f(x) — Ag(x) with respect to x, again subject to À > 0. 

Finally, it is straightforward to extend the technique of Lagrange multipliers to 
the case of multiple equality and inequality constraints. Suppose we wish to maxi- 
mize f(x) subject to g;(x) = 0 for j = 1,..., J, and h(x) > 0 fork = 1,..., K. 
We then introduce Lagrange multipliers {A;} and {ux}, and then optimize the La- 
grangian function given by 


J K 
L(x, {Ahia AA hke) Œ) 
j=1 k=1 


subject to ug > 0 and ughg(x) = 0 for k = 1,..., K. Extensions to constrained 
functional derivatives are similarly straightforward. For a more detailed discussion 
of the technique of Lagrange multipliers, see Nocedal and Wright (1999). 


References 


Abramowitz, M. and I. A. Stegun (1965). Handbook 
of Mathematical Functions. Dover. 


Adler, S. L. (1981). Over-relaxation method for the 
Monte Carlo evaluation of the partition func- 
tion for multiquadratic actions. Physical Review 
D 23, 2901-2904. 


Ahn, J. H. and J. H. Oh (2003). A constrained EM 
algorithm for principal component analysis. Neu- 
ral Computation 15(1), 57-65. 


Aizerman, M. A., E. M. Braverman, and L. I. Rozo- 
noer (1964). The probability problem of pattern 
recognition learning and the method of potential 


functions. Automation and Remote Control 25, 
1175-1190. 


Akaike, H. (1974). A new look at statistical model 
identification. JEEE Transactions on Automatic 
Control 19, 716-723. 


Ali, S. M. and S. D. Silvey (1966). A general class 
of coefficients of divergence of one distribution 
from another. Journal of the Royal Statistical So- 
ciety, B 28(1), 131-142. 

Allwein, E. L., R. E. Schapire, and Y. Singer (2000). 
Reducing multiclass to binary: a unifying ap- 
proach for margin classifiers. Journal of Machine 
Learning Research 1, 113-141. 


Amari, S. (1985). Differential-Geometrical Methods 
in Statistics. Springer. 


REFERENCES 711 


Amari, S., A. Cichocki, and H. H. Yang (1996). A 
new learning algorithm for blind signal separa- 
tion. In D. S. Touretzky, M. C. Mozer, and M. E. 
Hasselmo (Eds.), Advances in Neural Informa- 
tion Processing Systems, Volume 8, pp. 757-763. 
MIT Press. 


Amari, S. I. (1998). Natural gradient works effi- 
ciently in learning. Neural Computation 10, 
251-276. 


Anderson, J. A. and E. Rosenfeld (Eds.) (1988). 
Neurocomputing: Foundations of Research. MIT 
Press. 


Anderson, T. W. (1963). Asymptotic theory for prin- 
cipal component analysis. Annals of Mathemati- 
cal Statistics 34, 122-148. 


Andrieu, C., N. de Freitas, A. Doucet, and M. I. Jor- 
dan (2003). An introduction to MCMC for ma- 
chine learning. Machine Learning 50, 5—43. 


Anthony, M. and N. Biggs (1992). An Introduction 
to Computational Learning Theory. Cambridge 
University Press. 


Attias, H. (1999a). Independent factor analysis. Neu- 
ral Computation 11(4), 803-851. 


Attias, H. (1999b). Inferring parameters and struc- 
ture of latent variable models by variational 
Bayes. In K. B. Laskey and H. Prade (Eds.), 


712 REFERENCES 


Uncertainty in Artificial Intelligence: Proceed- 
ings of the Fifth Conference, pp. 21-30. Morgan 
Kaufmann. 


Bach, F. R. and M. I. Jordan (2002). Kernel inde- 
pendent component analysis. Journal of Machine 
Learning Research 3, 1—48. 


Bakir, G. H., J. Weston, and B. Scholkopf (2004). 
Learning to find pre-images. In S. Thrun, L. K. 
Saul, and B. Scholkopf (Eds.), Advances in Neu- 
ral Information Processing Systems, Volume 16, 
pp. 449-456. MIT Press. 


Baldi, P. and S. Brunak (2001). Bioinformatics: The 
Machine Learning Approach (Second ed.). MIT 
Press. 


Baldi, P. and K. Hornik (1989). Neural networks 
and principal component analysis: learning from 
examples without local minima. Neural Net- 
works 2(1), 53-58. 


Barber, D. and C. M. Bishop (1997). Bayesian 
model comparison by Monte Carlo chaining. In 
M. Mozer, M. Jordan, and T. Petsche (Eds.), Ad- 
vances in Neural Information Processing Sys- 
tems, Volume 9, pp. 333-339. MIT Press. 


Barber, D. and C. M. Bishop (1998a). Ensemble 
learning for multi-layer networks. In M. I. Jor- 
dan, K. J. Kearns, and S. A. Solla (Eds.), Ad- 
vances in Neural Information Processing Sys- 
tems, Volume 10, pp. 395—401. 

Barber, D. and C. M. Bishop (1998b). Ensemble 
learning in Bayesian neural networks. In C. M. 
Bishop (Ed.), Generalization in Neural Networks 
and Machine Learning, pp. 215-237. Springer. 

Bartholomew, D. J. (1987). Latent Variable Models 
and Factor Analysis. Charles Griffin. 

Basilevsky, A. (1994). Statistical Factor Analysis 
and Related Methods: Theory and Applications. 
Wiley. 

Bather, J. (2000). Decision Theory: An Introduction 
to Dynamic Programming and Sequential Deci- 
sions. Wiley. 

Baudat, G. and F. Anouar (2000). Generalized dis- 
criminant analysis using a kernel approach. Neu- 
ral Computation 12(10), 2385-2404. 


Baum, L. E. (1972). An inequality and associated 
maximization technique in statistical estimation 
of probabilistic functions of Markov processes. 
Inequalities 3, 1-8. 

Becker, S. and Y. Le Cun (1989). Improving the con- 
vergence of back-propagation learning with sec- 
ond order methods. In D. Touretzky, G. E. Hin- 
ton, and T. J. Sejnowski (Eds.), Proceedings of 
the 1988 Connectionist Models Summer School, 
pp. 29-37. Morgan Kaufmann. 


Bell, A. J. and T. J. Sejnowski (1995). An infor- 
mation maximization approach to blind separa- 
tion and blind deconvolution. Neural Computa- 
tion 7(6), 1129-1159. 


Bellman, R. (1961). Adaptive Control Processes: A 
Guided Tour. Princeton University Press. 


Bengio, Y. and P. Frasconi (1995). An input output 
HMM architecture. In G. Tesauro, D. S. Touret- 
zky, and T. K. Leen (Eds.), Advances in Neural 
Information Processing Systems, Volume 7, pp. 
427-434. MIT Press. 


Bennett, K. P. (1992). Robust linear programming 
discrimination of two linearly separable sets. Op- 
timization Methods and Software 1, 23-34. 


Berger, J. O. (1985). Statistical Decision Theory and 
Bayesian Analysis (Second ed.). Springer. 


Bernardo, J. M. and A. F. M. Smith (1994). Bayesian 
Theory. Wiley. 


Berrou, C., A. Glavieux, and P. Thitimajshima 
(1993). Near Shannon limit error-correcting cod- 
ing and decoding: Turbo-codes (1). In Proceed- 
ings ICC’93, pp. 1064-1070. 


Besag, J. (1974). On spatio-temporal models and 
Markov fields. In Transactions of the 7th Prague 
Conference on Information Theory, Statistical 
Decision Functions and Random Processes, pp. 
47-75. Academia. 


Besag, J. (1986). On the statistical analysis of dirty 
pictures. Journal of the Royal Statistical Soci- 
ety B-48, 259-302. 

Besag, J., P. J. Green, D. Hidgon, and K. Megersen 
(1995). Bayesian computation and stochastic 
systems. Statistical Science 10(1), 3-66. 


Bishop, C. M. (1991). A fast procedure for retraining 
the multilayer perceptron. International Journal 
of Neural Systems 2(3), 229-236. 


Bishop, C. M. (1992). Exact calculation of the Hes- 
sian matrix for the multilayer perceptron. Neural 
Computation 4(4), 494-501. 


Bishop, C. M. (1993). Curvature-driven smoothing: 
a learning algorithm for feedforward networks. 
IEEE Transactions on Neural Networks 4(5), 
882-884. 


Bishop, C. M. (1994). Novelty detection and neu- 
ral network validation. JEE Proceedings: Vision, 
Image and Signal Processing 141(4), 217-222. 
Special issue on applications of neural networks. 


Bishop, C. M. (1995a). Neural Networks for Pattern 
Recognition. Oxford University Press. 


Bishop, C. M. (1995b). Training with noise is equiv- 
alent to Tikhonov regularization. Neural Compu- 
tation 7(1), 108-116. 


Bishop, C. M. (1999a). Bayesian PCA. In M. S. 
Kearns, S. A. Solla, and D. A. Cohn (Eds.), Ad- 
vances in Neural Information Processing Sys- 
tems, Volume 11, pp. 382-388. MIT Press. 


Bishop, C. M. (1999b). Variational principal 
components. In Proceedings Ninth Interna- 


tional Conference on Artificial Neural Networks, 
ICANN’99, Volume 1, pp. 509-514. IEE. 


Bishop, C. M. and G. D. James (1993). Analysis of 
multiphase flows using dual-energy gamma den- 
sitometry and neural networks. Nuclear Instru- 
ments and Methods in Physics Research A327, 
580-593. 


Bishop, C. M. and I. T. Nabney (1996). Modelling 
conditional probability distributions for periodic 
variables. Neural Computation 8(5), 1123-1133. 


Bishop, C. M. and I. T. Nabney (2008). Pattern 
Recognition and Machine Learning: A Matlab 
Companion. Springer. In preparation. 


Bishop, C. M., D. Spiegelhalter, and J. Winn 
(2003). VIBES: A variational inference engine 
for Bayesian networks. In S. Becker, S. Thrun, 
and K. Obermeyer (Eds.), Advances in Neural 


REFERENCES 713 


Information Processing Systems, Volume 15, pp. 
793-800. MIT Press. 


Bishop, C. M. and M. Svensén (2003). Bayesian hi- 
erarchical mixtures of experts. In U. Kjaerulff 
and C. Meek (Eds.), Proceedings Nineteenth 
Conference on Uncertainty in Artificial Intelli- 
gence, pp. 57-64. Morgan Kaufmann. 


Bishop, C. M., M. Svensén, and G. E. Hinton 
(2004). Distinguishing text from graphics in on- 
line handwritten ink. In F. Kimura and H. Fu- 
jisawa (Eds.), Proceedings Ninth International 
Workshop on Frontiers in Handwriting Recogni- 
tion, IWFHR-9, Tokyo, Japan, pp. 142-147. 


Bishop, C. M., M. Svensén, and C. K. I. Williams 
(1996). EM optimization of latent variable den- 
sity models. In D. S. Touretzky, M. C. Mozer, 
and M. E. Hasselmo (Eds.), Advances in Neural 
Information Processing Systems, Volume 8, pp. 
465-471. MIT Press. 


Bishop, C. M., M. Svensén, and C. K. I. Williams 
(1997a). GTM: a principled alternative to the 
Self-Organizing Map. In M. C. Mozer, M. I. Jor- 
dan, and T. Petche (Eds.), Advances in Neural 
Information Processing Systems, Volume 9, pp. 
354-360. MIT Press. 


Bishop, C. M., M. Svensén, and C. K. I. Williams 
(1997b). Magnification factors for the GTM al- 
gorithm. In Proceedings IEE Fifth International 
Conference on Artificial Neural Networks, Cam- 
bridge, U.K., pp. 64—69. Institute of Electrical 
Engineers. 


Bishop, C. M., M. Svensén, and C. K. I. Williams 
(1998a). Developments of the Generative To- 
pographic Mapping. Neurocomputing 21, 203- 
224. 


Bishop, C. M., M. Svensén, and C. K. I. Williams 
(1998b). GTM: the Generative Topographic 
Mapping. Neural Computation 10(1), 215-234. 


Bishop, C. M. and M. E. Tipping (1998). A hier- 
archical latent variable model for data visualiza- 
tion. IEEE Transactions on Pattern Analysis and 
Machine Intelligence 20(3), 281-293. 


714 REFERENCES 


Bishop, C. M. and J. Winn (2000). Non-linear 
Bayesian image modelling. In Proceedings Sixth 
European Conference on Computer Vision, 
Dublin, Volume 1, pp. 3-17. Springer. 


Blei, D. M., M. I. Jordan, and A. Y. Ng (2003). Hi- 
erarchical Bayesian models for applications in 
information retrieval. In J. M. B. et al. (Ed.), 
Bayesian Statistics, 7, pp. 25-43. Oxford Uni- 
versity Press. 


Block, H. D. (1962). The perceptron: a model 
for brain functioning. Reviews of Modern 
Physics 34(1), 123-135. Reprinted in Anderson 
and Rosenfeld (1988). 


Blum, J. A. (1965). Multidimensional stochastic ap- 
proximation methods. Annals of Mathematical 
Statistics 25, 737-744. 


Bodlaender, H. (1993). A tourist guide through 
treewidth. Acta Cybernetica 11, 1-21. 


Boser, B. E., I. M. Guyon, and V. N. Vapnik (1992). 
A training algorithm for optimal margin classi- 
fiers. In D. Haussler (Ed.), Proceedings Fifth An- 
nual Workshop on Computational Learning The- 
ory (COLT), pp. 144-152. ACM. 


Bourlard, H. and Y. Kamp (1988). Auto-association 
by multilayer perceptrons and singular value de- 
composition. Biological Cybernetics 59, 291- 
294. 


Box, G. E. P., G. M. Jenkins, and G. C. Reinsel 
(1994). Time Series Analysis. Prentice Hall. 


Box, G. E. P. and G. C. Tao (1973). Bayesian Infer- 
ence in Statistical Analysis. Wiley. 


Boyd, S. and L. Vandenberghe (2004). Convex Opti- 
mization. Cambridge University Press. 


Boyen, X. and D. Koller (1998). Tractable inference 
for complex stochastic processes. In G. F. Cooper 
and S. Moral (Eds.), Proceedings 14th Annual 
Conference on Uncertainty in Artificial Intelli- 
gence (UAI), pp. 33-42. Morgan Kaufmann. 

Boykov, Y., O. Veksler, and R. Zabih (2001). Fast 
approximate energy minimization via graph cuts. 


IEEE Transactions on Pattern Analysis and Ma- 
chine Intelligence 23(11), 1222-1239. 


Breiman, L. (1996). Bagging predictors. Machine 
Learning 26, 123-140. 


Breiman, L., J. H. Friedman, R. A. Olshen, and 
P. J. Stone (1984). Classification and Regression 
Trees. Wadsworth. 


Brooks, S. P. (1998). Markov chain Monte 
Carlo method and its application. The Statisti- 
cian 47(1), 69-100. 


Broomhead, D. S. and D. Lowe (1988). Multivari- 
able functional interpolation and adaptive net- 
works. Complex Systems 2, 321-355. 


Buntine, W. and A. Weigend (1991). Bayesian back- 
propagation. Complex Systems 5, 603-643. 


Buntine, W. L. and A. S. Weigend (1993). Com- 
puting second derivatives in feed-forward net- 
works: a review. IEEE Transactions on Neural 
Networks 5(3), 480-488. 


Burges, C. J. C. (1998). A tutorial on support vec- 
tor machines for pattern recognition. Knowledge 
Discovery and Data Mining 2(2), 121-167. 


Cardoso, J.-F. (1998). Blind signal separation: statis- 
tical principles. Proceedings of the IEEE 9(10), 
2009-2025. 

Casella, G. and R. L. Berger (2002). Statistical In- 
ference (Second ed.). Duxbury. 

Castillo, E., J. M. Gutiérrez, and A. S. Hadi (1997). 
Expert Systems and Probabilistic Network Mod- 
els. Springer. 

Chan, K., T. Lee, and T. J. Sejnowski (2003). Vari- 
ational Bayesian learning of ICA with missing 
data. Neural Computation 15(8), 1991-2011. 

Chen, A. M., H. Lu, and R. Hecht-Nielsen (1993). 
On the geometry of feedforward neural network 
error surfaces. Neural Computation 5(6), 910- 
927. 

Chen, M. H., Q. M. Shao, and J. G. Ibrahim (Eds.) 
(2001). Monte Carlo Methods for Bayesian Com- 
putation. Springer. 

Chen, S., C. F. N. Cowan, and P. M. Grant (1991). 
Orthogonal least squares learning algorithm for 


radial basis function networks. IEEE Transac- 
tions on Neural Networks 2(2), 302-309. 


Choudrey, R. A. and S. J. Roberts (2003). Variational 
mixture of Bayesian independent component an- 
alyzers. Neural Computation 15(1), 213-252. 


Clifford, P. (1990). Markov random fields in statis- 
tics. In G. R. Grimmett and D. J. A. Welsh (Eds.), 
Disorder in Physical Systems. A Volume in Hon- 
our of John M. Hammersley, pp. 19-32. Oxford 
University Press. 


Collins, M., S. Dasgupta, and R. E. Schapire (2002). 
A generalization of principal component analy- 
sis to the exponential family. In T. G. Dietterich, 
S. Becker, and Z. Ghahramani (Eds.), Advances 
in Neural Information Processing Systems, Vol- 
ume 14, pp. 617—624. MIT Press. 


Comon, P., C. Jutten, and J. Herault (1991). Blind 
source separation, 2: problems statement. Signal 
Processing 24(1), 11-20. 


Corduneanu, A. and C. M. Bishop (2001). Vari- 
ational Bayesian model selection for mixture 
distributions. In T. Richardson and T. Jaakkola 
(Eds.), Proceedings Eighth International Confer- 
ence on Artificial Intelligence and Statistics, pp. 
27-34. Morgan Kaufmann. 


Cormen, T. H., C. E. Leiserson, R. L. Rivest, and 
C. Stein (2001). Introduction to Algorithms (Sec- 
ond ed.). MIT Press. 


Cortes, C. and V. N. Vapnik (1995). Support vector 
networks. Machine Learning 20, 273-297. 


Cotter, N. E. (1990). The Stone-Weierstrass theo- 
rem and its application to neural networks. IEEE 
Transactions on Neural Networks 1(4), 290-295. 


Cover, T. and P. Hart (1967). Nearest neighbor pat- 
tern classification. IEEE Transactions on Infor- 
mation Theory IT-11, 21-27. 


Cover, T. M. and J. A. Thomas (1991). Elements of 
Information Theory. Wiley. 


Cowell, R. G., A. P. Dawid, S. L. Lauritzen, and D. J. 
Spiegelhalter (1999). Probabilistic Networks and 
Expert Systems. Springer. 


Cox, R. T. (1946). Probability, frequency and 
reasonable expectation. American Journal of 
Physics 14(1), 1-13. 


REFERENCES 715 


Cox, T. F. and M. A. A. Cox (2000). Multidimen- 
sional Scaling (Second ed.). Chapman and Hall. 


Cressie, N. (1993). Statistics for Spatial Data. Wiley. 


Cristianini, N. and J. Shawe-Taylor (2000). Support 
vector machines and other kernel-based learning 
methods. Cambridge University Press. 


Csató, L. and M. Opper (2002). Sparse on-line Gaus- 
sian processes. Neural Computation 14(3), 641- 
668. 


Csiszar, I. and G. Tusnady (1984). Information ge- 
ometry and alternating minimization procedures. 
Statistics and Decisions 1(1), 205-237. 

Cybenko, G. (1989). Approximation by superposi- 
tions of a sigmoidal function. Mathematics of 
Control, Signals and Systems 2, 304-314. 

Dawid, A. P. (1979). Conditional independence in 
statistical theory (with discussion). Journal of the 
Royal Statistical Society, Series B 4, 1-31. 

Dawid, A. P. (1980). Conditional independence for 


statistical operations. Annals of Statistics 8, 598— 
617. 


deFinetti, B. (1970). Theory of Probability. Wiley 
and Sons. 


Dempster, A. P, N. M. Laird, and D. B. Rubin 
(1977). Maximum likelihood from incomplete 
data via the EM algorithm. Journal of the Royal 
Statistical Society, B 39(1), 1-38. 

Denison, D. G. T., C. C. Holmes, B. K. Mallick, 


and A. F. M. Smith (2002). Bayesian Methods for 
Nonlinear Classification and Regression. Wiley. 


Diaconis, P. and L. Saloff-Coste (1998). What do we 
know about the Metropolis algorithm? Journal 
of Computer and System Sciences 57, 20-36. 

Dietterich, T. G. and G. Bakiri (1995). Solving 
multiclass learning problems via error-correcting 
output codes. Journal of Artificial Intelligence 
Research 2, 263-286. 

Duane, S., A. D. Kennedy, B. J. Pendleton, and 
D. Roweth (1987). Hybrid Monte Carlo. Physics 
Letters B 195(2), 216-222. 

Duda, R. O. and P. E. Hart (1973). Pattern Classifi- 
cation and Scene Analysis. Wiley. 


716 REFERENCES 


Duda, R. O., P. E. Hart, and D. G. Stork (2001). Pat- 
tern Classification (Second ed.). Wiley. 


Durbin, R., S. Eddy, A. Krogh, and G. Mitchi- 
son (1998). Biological Sequence Analysis. Cam- 
bridge University Press. 


Dybowski, R. and S. Roberts (2005). An anthology 
of probabilistic models for medical informatics. 
In D. Husmeier, R. Dybowski, and S. Roberts 
(Eds.), Probabilistic Modeling in Bioinformatics 
and Medical Informatics, pp. 297-349. Springer. 


Efron, B. (1979). Bootstrap methods: another look 
at the jackknife. Annals of Statistics 7, 1-26. 


Elkan, C. (2003). Using the triangle inequality to ac- 
celerate k-means. In Proceedings of the Twelfth 


International Conference on Machine Learning, 
pp. 147-153. AAAT. 


Elliott, R. J., L. Aggoun, and J. B. Moore (1995). 
Hidden Markov Models: Estimation and Con- 
trol. Springer. 


Ephraim, Y., D. Malah, and B. H. Juang (1989). 
On the application of hidden Markov models for 
enhancing noisy speech. JEEE Transactions on 
Acoustics, Speech and Signal Processing 37(12), 
1846-1856. 


Erwin, E., K. Obermayer, and K. Schulten (1992). 
Self-organizing maps: ordering, convergence 
properties and energy functions. Biological Cy- 
bernetics 67, 47-55. 


Everitt, B. S. (1984). An Introduction to Latent Vari- 
able Models. Chapman and Hall. 


Faul, A. C. and M. E. Tipping (2002). Analysis of 
sparse Bayesian learning. In T. G. Dietterich, 
S. Becker, and Z. Ghahramani (Eds.), Advances 
in Neural Information Processing Systems, Vol- 
ume 14, pp. 383-389. MIT Press. 


Feller, W. (1966). An Introduction to Probability 
Theory and its Applications (Second ed.), Vol- 
ume 2. Wiley. 


Feynman, R. P, R. B. Leighton, and M. Sands 
(1964). The Feynman Lectures of Physics, Vol- 
ume Two. Addison-Wesley. Chapter 19. 


Fletcher, R. (1987). Practical Methods of Optimiza- 
tion (Second ed.). Wiley. 


Forsyth, D. A. and J. Ponce (2003). Computer Vi- 
sion: A Modern Approach. Prentice Hall. 


Freund, Y. and R. E. Schapire (1996). Experiments 
with a new boosting algorithm. In L. Saitta (Ed.), 
Thirteenth International Conference on Machine 
Learning, pp. 148—156. Morgan Kaufmann. 


Frey, B. J. (1998). Graphical Models for Ma- 
chine Learning and Digital Communication. 
MIT Press. 


Frey, B. J. and D. J. C. MacKay (1998). A revolu- 
tion: Belief propagation in graphs with cycles. In 
M. I. Jordan, M. J. Kearns, and S. A. Solla (Eds.), 
Advances in Neural Information Processing Sys- 
tems, Volume 10. MIT Press. 


Friedman, J. H. (2001). Greedy function approxi- 
mation: a gradient boosting machine. Annals of 
Statistics 29(5), 1189-1232. 


Friedman, J. H., T. Hastie, and R. Tibshirani (2000). 
Additive logistic regression: a statistical view of 
boosting. Annals of Statistics 28, 337-407. 


Friedman, N. and D. Koller (2003). Being Bayesian 
about network structure: A Bayesian approach 
to structure discovery in Bayesian networks. Ma- 
chine Learning 50, 95-126. 


Frydenberg, M. (1990). The chain graph Markov 
property. Scandinavian Journal of Statistics 17, 
333-353. 


Fukunaga, K. (1990). Introduction to Statistical Pat- 
tern Recognition (Second ed.). Academic Press. 


Funahashi, K. (1989). On the approximate realiza- 
tion of continuous mappings by neural networks. 
Neural Networks 2(3), 183-192. 


Fung, R. and K. C. Chang (1990). Weighting and 
integrating evidence for stochastic simulation in 
Bayesian networks. In P. P. Bonissone, M. Hen- 
rion, L. N. Kanal, and J. F. Lemmer (Eds.), Un- 
certainty in Artificial Intelligence, Volume 5, pp. 
208-219. Elsevier. 


Gallager, R. G. (1963). Low-Density Parity-Check 
Codes. MIT Press. 


Gamerman, D. (1997). Markov Chain Monte Carlo: 
Stochastic Simulation for Bayesian Inference. 
Chapman and Hall. 


Gelman, A., J. B. Carlin, H. S. Stern, and D. B. Ru- 
bin (2004). Bayesian Data Analysis (Second ed.). 
Chapman and Hall. 


Geman, S. and D. Geman (1984). Stochastic re- 
laxation, Gibbs distributions, and the Bayesian 
restoration of images. IEEE Transactions on Pat- 


tern Analysis and Machine Intelligence 6(1), 
721-741. 


Ghahramani, Z. and M. J. Beal (2000). Variational 
inference for Bayesian mixtures of factor ana- 
lyzers. In S. A. Solla, T. K. Leen, and K. R. 
Muller (Eds.), Advances in Neural Information 
Processing Systems, Volume 12, pp. 449-455. 
MIT Press. 


Ghahramani, Z. and G. E. Hinton (1996a). The 
EM algorithm for mixtures of factor analyzers. 
Technical Report CRG-TR-96-1, University of 
Toronto. 


Ghahramani, Z. and G. E. Hinton (1996b). Param- 
eter estimation for linear dynamical systems. 
Technical Report CRG-TR-96-2, University of 
Toronto. 


Ghahramani, Z. and G. E. Hinton (1998). Variational 
learning for switching state-space models. Neu- 
ral Computation 12(4), 963-996. 


Ghahramani, Z. and M. I. Jordan (1994). Super- 
vised learning from incomplete data via an EM 
appproach. In J. D. Cowan, G. T. Tesauro, and 
J. Alspector (Eds.), Advances in Neural Informa- 
tion Processing Systems, Volume 6, pp. 120-127. 
Morgan Kaufmann. 

Ghahramani, Z. and M. I. Jordan (1997). Factorial 
hidden Markov models. Machine Learning 29, 
245-275. 

Gibbs, M. N. (1997). Bayesian Gaussian processes 
for regression and classification. Phd thesis, Uni- 
versity of Cambridge. 

Gibbs, M. N. and D. J. C. MacKay (2000). Varia- 
tional Gaussian process classifiers. JEEE Trans- 
actions on Neural Networks 11, 1458-1464. 


REFERENCES 717 


Gilks, W. R. (1992). Derivative-free adaptive 
rejection sampling for Gibbs sampling. In 
J. Bernardo, J. Berger, A. P. Dawid, and A. F. M. 
Smith (Eds.), Bayesian Statistics, Volume 4. Ox- 
ford University Press. 


Gilks, W. R., N. G. Best, and K. K. C. Tan (1995). 
Adaptive rejection Metropolis sampling. Applied 
Statistics 44, 455-472. 


Gilks, W. R., S. Richardson, and D. J. Spiegelhal- 
ter (Eds.) (1996). Markov Chain Monte Carlo in 
Practice. Chapman and Hall. 


Gilks, W. R. and P. Wild (1992). Adaptive rejection 
sampling for Gibbs sampling. Applied Statis- 
tics 41, 337-348. 


Gill, P. E., W. Murray, and M. H. Wright (1981). 
Practical Optimization. Academic Press. 


Goldberg, P. W., C. K. I. Williams, and C. M. 
Bishop (1998). Regression with input-dependent 
noise: A Gaussian process treatment. In Ad- 
vances in Neural Information Processing Sys- 
tems, Volume 10, pp. 493-499. MIT Press. 


Golub, G. H. and C. F. Van Loan (1996). Matrix 
Computations (Third ed.). John Hopkins Univer- 
sity Press. 


Good, I. (1950). Probability and the Weighing of Ev- 
idence. Hafners. 


Gordon, N. J., D. J. Salmond, and A. F. M. Smith 
(1993). Novel approach to  nonlinear/non- 
Gaussian Bayesian state estimation. IEE 
Proceedings-F 140(2), 107-113. 


Graepel, T. (2003). Solving noisy linear operator 
equations by Gaussian processes: Application 
to ordinary and partial differential equations. In 
Proceedings of the Twentieth International Con- 
ference on Machine Learning, pp. 234-241. 


Greig, D., B. Porteous, and A. Seheult (1989). Ex- 
act maximum a-posteriori estimation for binary 
images. Journal of the Royal Statistical Society, 
Series B 51(2), 271-279. 

Gull, S. F. (1989). Developments in maximum en- 
tropy data analysis. In J. Skilling (Ed.), Maxi- 
mum Entropy and Bayesian Methods, pp. 53-71. 
Kluwer. 


718 REFERENCES 


Hassibi, B. and D. G. Stork (1993). Second order 
derivatives for network pruning: optimal brain 
surgeon. In S. J. Hanson, J. D. Cowan, and 
C. L. Giles (Eds.), Advances in Neural Informa- 
tion Processing Systems, Volume 5, pp. 164-171. 
Morgan Kaufmann. 


Hastie, T. and W. Stuetzle (1989). Principal curves. 
Journal of the American Statistical Associa- 
tion 84(106), 502-516. 


Hastie, T., R. Tibshirani, and J. Friedman (2001). 
The Elements of Statistical Learning. Springer. 


Hastings, W. K. (1970). Monte Carlo sampling 
methods using Markov chains and their applica- 
tions. Biometrika 57, 97-109. 


Hathaway, R. J. (1986). Another interpretation of the 
EM algorithm for mixture distributions. Statistics 
and Probability Letters 4, 53—56. 


Haussler, D. (1999). Convolution kernels on discrete 
structures. Technical Report UCSC-CRL-99-10, 
University of California, Santa Cruz, Computer 
Science Department. 


Henrion, M. (1988). Propagation of uncertainty by 
logic sampling in Bayes’ networks. In J. F. Lem- 
mer and L. N. Kanal (Eds.), Uncertainty in Arti- 
ficial Intelligence, Volume 2, pp. 149-164. North 
Holland. 


Herbrich, R. (2002). Learning Kernel Classifiers. 
MIT Press. 


Hertz, J., A. Krogh, and R. G. Palmer (1991). In- 
troduction to the Theory of Neural Computation. 
Addison Wesley. 


Hinton, G. E., P. Dayan, and M. Revow (1997). 
Modelling the manifolds of images of handwrit- 
ten digits. IEEE Transactions on Neural Net- 
works 8(1), 65—74. 


Hinton, G. E. and D. van Camp (1993). Keeping 
neural networks simple by minimizing the de- 
scription length of the weights. In Proceedings of 
the Sixth Annual Conference on Computational 
Learning Theory, pp. 5-13. ACM. 


Hinton, G. E., M. Welling, Y. W. Teh, and S. Osin- 
dero (2001). A new view of ICA. In Proceedings 


of the International Conference on Independent 
Component Analysis and Blind Signal Separa- 
tion, Volume 3. 


Hodgson, M. E. (1998). Reducing computational re- 
quirements of the minimum-distance classifier. 
Remote Sensing of Environments 25, 117-128. 


Hoerl, A. E. and R. Kennard (1970). Ridge regres- 
sion: biased estimation for nonorthogonal prob- 
lems. Technometrics 12, 55—67. 


Hofmann, T. (2000). Learning the similarity of doc- 
uments: an information-geometric approach to 
document retrieval and classification. In S. A. 
Solla, T. K. Leen, and K. R. Muller (Eds.), Ad- 
vances in Neural Information Processing Sys- 
tems, Volume 12, pp. 914-920. MIT Press. 


Hojen-Sorensen, P. A., O. Winther, and L. K. Hansen 
(2002). Mean field approaches to independent 
component analysis. Neural Computation 14(4), 
889-918. 


Hornik, K. (1991). Approximation capabilities of 
multilayer feedforward networks. Neural Net- 
works 4(2), 251-257. 


Hornik, K., M. Stinchcombe, and H. White (1989). 
Multilayer feedforward networks are universal 
approximators. Neural Networks 2(5), 359-366. 


Hotelling, H. (1933). Analysis of a complex of statis- 
tical variables into principal components. Jour- 
nal of Educational Psychology 24, 417-441. 


Hotelling, H. (1936). Relations between two sets of 
variables. Biometrika 28, 321-377. 


Hyvarinen, A. and E. Oja (1997). A fast fixed-point 
algorithm for independent component analysis. 
Neural Computation 9(7), 1483-1492. 


Isard, M. and A. Blake (1998). CONDENSATION 
— conditional density propagation for visual 
tracking. International Journal of Computer Vi- 
sion 29(1), 5-18. 


Ito, Y. (1991). Representation of functions by su- 
perpositions of a step or sigmoid function and 
their applications to neural network theory. Neu- 
ral Networks 4(3), 385-394. 


Jaakkola, T. and M. I. Jordan (2000). Bayesian 
parameter estimation via variational methods. 
Statistics and Computing 10, 25-37. 


Jaakkola, T. S. (2001). Tutorial on variational ap- 
proximation methods. In M. Opper and D. Saad 
(Eds.), Advances in Mean Field Methods, pp. 
129-159. MIT Press. 


Jaakkola, T. S. and D. Haussler (1999). Exploiting 
generative models in discriminative classifiers. In 
M. S. Kearns, S. A. Solla, and D. A. Cohn (Eds.), 
Advances in Neural Information Processing Sys- 
tems, Volume 11. MIT Press. 


Jacobs, R. A., M. I. Jordan, S. J. Nowlan, and G. E. 
Hinton (1991). Adaptive mixtures of local ex- 
perts. Neural Computation 3(1), 79-87. 


Jaynes, E. T. (2003). Probability Theory: The Logic 
of Science. Cambridge University Press. 


Jebara, T. (2004). Machine Learning: Discrimina- 
tive and Generative. Kluwer. 


Jeffries, H. (1946). An invariant form for the prior 
probability in estimation problems. Pro. Roy. 
Soc. AA 186, 453-461. 


Jelinek, F. (1997). Statistical Methods for Speech 
Recognition. MIT Press. 


Jensen, C., A. Kong, and U. Kjaerulff (1995). Block- 
ing gibbs sampling in very large probabilistic 
expert systems. International Journal of Human 
Computer Studies. Special Issue on Real-World 
Applications of Uncertain Reasoning. 42, 647- 
666. 


Jensen, F. V. (1996). An Introduction to Bayesian 
Networks. UCL Press. 


Jerrum, M. and A. Sinclair (1996). The Markov 
chain Monte Carlo method: an approach to ap- 
proximate counting and integration. In D. S. 
Hochbaum (Ed.), Approximation Algorithms for 
NP-Hard Problems. PWS Publishing. 


Jolliffe, I. T. (2002). Principal Component Analysis 
(Second ed.). Springer. 


Jordan, M. I. (1999). Learning in Graphical Models. 
MIT Press. 


REFERENCES 719 


Jordan, M. I. (2007). An Introduction to Probabilis- 
tic Graphical Models. In preparation. 


Jordan, M. I., Z. Ghahramani, T. S. Jaakkola, and 
L. K. Saul (1999). An introduction to variational 
methods for graphical models. In M. I. Jordan 
(Ed.), Learning in Graphical Models, pp. 105- 
162. MIT Press. 


Jordan, M. I. and R. A. Jacobs (1994). Hierarchical 
mixtures of experts and the EM algorithm. Neu- 
ral Computation 6(2), 181-214. 


Jutten, C. and J. Herault (1991). Blind separation of 
sources, 1: An adaptive algorithm based on neu- 
romimetic architecture. Signal Processing 24(1), 
1-10. 


Kalman, R. E. (1960). A new approach to linear fil- 
tering and prediction problems. Transactions of 
the American Society for Mechanical Engineer- 
ing, Series D, Journal of Basic Engineering 82, 
35-45. 


Kambhatla, N. and T. K. Leen (1997). Dimension 
reduction by local principal component analysis. 
Neural Computation 9(7), 1493-1516. 


Kanazawa, K., D. Koller, and S. Russel (1995). 
Stochastic simulation algorithms for dynamic 
probabilistic networks. In Uncertainty in Artifi- 
cial Intelligence, Volume 11. Morgan Kaufmann. 


Kapadia, S. (1998). Discriminative Training of Hid- 
den Markov Models. Phd thesis, University of 
Cambridge, U.K. 


Kapur, J. (1989). Maximum entropy methods in sci- 
ence and engineering. Wiley. 


Karush, W. (1939). Minima of functions of several 
variables with inequalities as side constraints. 
Master’s thesis, Department of Mathematics, 
University of Chicago. 


Kass, R. E. and A. E. Raftery (1995). Bayes fac- 
tors. Journal of the American Statistical Associ- 
ation 90, 377-395. 


Kearns, M. J. and U. V. Vazirani (1994). An Intro- 
duction to Computational Learning Theory. MIT 
Press. 


720 REFERENCES 


Kindermann, R. and J. L. Snell (1980). Markov Ran- 
dom Fields and Their Applications. American 
Mathematical Society. 


Kittler, J. and J. Foglein (1984). Contextual classifi- 
cation of multispectral pixel data. Image and Vi- 
sion Computing 2, 13-29. 


Kohonen, T. (1982). Self-organized formation of 
topologically correct feature maps. Biological 
Cybernetics 43, 59-69. 


Kohonen, T. (1995). 
Springer. 


Kolmogorov, V. and R. Zabih (2004). What en- 
ergy functions can be minimized via graph cuts? 
IEEE Transactions on Pattern Analysis and Ma- 
chine Intelligence 26(2), 147-159. 


Kreinovich, V. Y. (1991). Arbitrary nonlinearity is 
sufficient to represent all functions by neural net- 
works: a theorem. Neural Networks 4(3), 381-— 
383. 


Krogh, A., M. Brown, I. S. Mian, K. Sjölander, and 
D. Haussler (1994). Hidden Markov models in 
computational biology: Applications to protein 
modelling. Journal of Molecular Biology 235, 
1501-1531. 


Kschischnang, F. R., B. J. Frey, and H. A. Loeliger 
(2001). Factor graphs and the sum-product algo- 
rithm. JEEE Transactions on Information The- 
ory 47(2), 498-519. 


Kuhn, H. W. and A. W. Tucker (1951). Nonlinear 
programming. In Proceedings of the 2nd Berke- 
ley Symposium on Mathematical Statistics and 
Probabilities, pp. 481—492. University of Cali- 
fornia Press. 


Kullback, S. and R. A. Leibler (1951). On infor- 
mation and sufficiency. Annals of Mathematical 
Statistics 22(1), 79—86. 


Kurkova, V. and P. C. Kainen (1994). Functionally 
equivalent feed-forward neural networks. Neural 
Computation 6(3), 543-558. 


Kuss, M. and C. Rasmussen (2006). Assessing ap- 
proximations for Gaussian process classification. 


Self-Organizing Maps. 


In Advances in Neural Information Processing 
Systems, Number 18. MIT Press. in press. 


Lasserre, J., C. M. Bishop, and T. Minka (2006). 
Principled hybrids of generative and discrimina- 
tive models. In Proceedings 2006 IEEE Confer- 
ence on Computer Vision and Pattern Recogni- 
tion, New York. 


Lauritzen, S. and N. Wermuth (1989). Graphical 
models for association between variables, some 
of which are qualitative some quantitative. An- 
nals of Statistics 17, 31-57. 


Lauritzen, S. L. (1992). Propagation of probabilities, 
means and variances in mixed graphical associa- 
tion models. Journal of the American Statistical 
Association 87, 1098-1108. 


Lauritzen, S. L. (1996). Graphical Models. Oxford 
University Press. 


Lauritzen, S. L. and D. J. Spiegelhalter (1988). Lo- 
cal computations with probabailities on graphical 
structures and their application to expert systems. 
Journal of the Royal Statistical Society 50, 157- 
224. 


Lawley, D. N. (1953). A modified method of esti- 
mation in factor analysis and some large sam- 
ple results. In Uppsala Symposium on Psycho- 
logical Factor Analysis, Number 3 in Nordisk 
Psykologi Monograph Series, pp. 35—42. Upp- 
sala: Almqvist and Wiksell. 


Lawrence, N. D., A. I. T. Rowstron, C. M. Bishop, 
and M. J. Taylor (2002). Optimising synchro- 
nisation times for mobile devices. In T. G. Di- 
etterich, S. Becker, and Z. Ghahramani (Eds.), 
Advances in Neural Information Processing Sys- 
tems, Volume 14, pp. 1401-1408. MIT Press. 

Lazarsfeld, P. F. and N. W. Henry (1968). Latent 
Structure Analysis. Houghton Mifflin. 

Le Cun, Y., B. Boser, J. S. Denker, D. Henderson, 
R. E. Howard, W. Hubbard, and L. D. Jackel 
(1989). Backpropagation applied to handwritten 
zip code recognition. Neural Computation 1(4), 
541-551. 

Le Cun, Y., J. S. Denker, and S. A. Solla (1990). 
Optimal brain damage. In D. S. Touretzky (Ed.), 


Advances in Neural Information Processing Sys- 
tems, Volume 2, pp. 598-605. Morgan Kauf- 
mann. 


LeCun, Y., L. Bottou, Y. Bengio, and P. Haffner 
(1998). Gradient-based learning applied to doc- 
ument recognition. Proceedings of the IEEE 86, 
2278-2324. 

Lee, Y., Y. Lin, and G. Wahba (2001). Multicategory 
support vector machines. Technical Report 1040, 
Department of Statistics, University of Madison, 
Wisconsin. 

Leen, T. K. (1995). From data distributions to regu- 
larization in invariant learning. Neural Computa- 
tion 7,974-981. 

Lindley, D. V. (1982). Scoring rules and the in- 
evitability of probability. International Statisti- 
cal Review 50, 1—26. 

Liu, J. S. (Ed.) (2001). Monte Carlo Strategies in 
Scientific Computing. Springer. 

Lloyd, S. P. (1982). Least squares quantization in 
PCM. IEEE Transactions on Information The- 
ory 28(2), 129-137. 


Lutkepohl, H. (1996). Handbook of Matrices. Wiley. 


MacKay, D. J. C. (1992a). Bayesian interpolation. 
Neural Computation 4(3), 415—447. 


MacKay, D. J. C. (1992b). The evidence framework 
applied to classification networks. Neural Com- 
putation 4(5), 720-736. 


MacKay, D. J. C. (1992c). A practical Bayesian 
framework for back-propagation networks. Neu- 
ral Computation 4(3), 448-472. 


MacKay, D. J. C. (1994). Bayesian methods for 
backprop networks. In E. Domany, J. L. van 
Hemmen, and K. Schulten (Eds.), Models of 
Neural Networks, III, Chapter 6, pp. 211-254. 
Springer. 

MacKay, D. J. C. (1995). Bayesian neural networks 
and density networks. Nuclear Instruments and 
Methods in Physics Research, A 354(1), 73-80. 


MacKay, D. J. C. (1997). Ensemble learning for hid- 
den Markov models. Unpublished manuscript, 


REFERENCES 721 


Department of Physics, University of Cam- 
bridge. 


MacKay, D. J. C. (1998). Introduction to Gaus- 
sian processes. In C. M. Bishop (Ed.), Neural 
Networks and Machine Learning, pp. 133-166. 
Springer. 

MacKay, D. J. C. (1999). Comparison of approx- 
imate methods for handling hyperparameters. 
Neural Computation 11(5), 1035-1068. 


MacKay, D. J. C. (2003). Information Theory, Infer- 
ence and Learning Algorithms. Cambridge Uni- 
versity Press. 


MacKay, D. J. C. and M. N. Gibbs (1999). Den- 
sity networks. In J. W. Kay and D. M. Tittering- 
ton (Eds.), Statistics and Neural Networks: Ad- 
vances at the Interface, Chapter 5, pp. 129-145. 
Oxford University Press. 


MacKay, D. J.C. and R. M. Neal (1999). Good error- 
correcting codes based on very sparse matrices. 
IEEE Transactions on Information Theory 45, 
399-431. 


MacQueen, J. (1967). Some methods for classifica- 
tion and analysis of multivariate observations. In 
L. M. LeCam and J. Neyman (Eds.), Proceed- 
ings of the Fifth Berkeley Symposium on Mathe- 
matical Statistics and Probability, Volume I, pp. 
281-297. University of California Press. 


Magnus, J. R. and H. Neudecker (1999). Matrix Dif- 
ferential Calculus with Applications in Statistics 
and Econometrics. Wiley. 


Mallat, S. (1999). A Wavelet Tour of Signal Process- 
ing (Second ed.). Academic Press. 


Manning, C. D. and H. Schutze (1999). Foundations 
of Statistical Natural Language Processing. MIT 
Press. 


Mardia, K. V. and P. E. Jupp (2000). Directional 
Statistics. Wiley. 

Maybeck, P. S. (1982). Stochastic models, estima- 
tion and control. Academic Press. 


McAllester, D. A. (2003). PAC-Bayesian stochastic 
model selection. Machine Learning 51(1), 5-21. 


722 REFERENCES 


McCullagh, P. and J. A. Nelder (1989). Generalized 
Linear Models (Second ed.). Chapman and Hall. 


McCulloch, W. S. and W. Pitts (1943). A logical 
calculus of the ideas immanent in nervous ac- 
tivity. Bulletin of Mathematical Biophysics 5, 
115-133. Reprinted in Anderson and Rosenfeld 
(1988). 


McEliece, R. J., D. J. C. MacKay, and J. F. Cheng 
(1998). Turbo decoding as an instance of Pearl’s 
‘Belief Ppropagation’ algorithm. IEEE Journal 
on Selected Areas in Communications 16, 140- 
152. 


McLachlan, G. J. and K. E. Basford (1988). Mixture 
Models: Inference and Applications to Cluster- 
ing. Marcel Dekker. 


McLachlan, G. J. and T. Krishnan (1997). The EM 
Algorithm and its Extensions. Wiley. 


McLachlan, G. J. and D. Peel (2000). Finite Mixture 
Models. Wiley. 


Meng, X. L. and D. B. Rubin (1993). Maximum like- 
lihood estimation via the ECM algorithm: a gen- 
eral framework. Biometrika 80, 267—278. 


Metropolis, N., A. W. Rosenbluth, M. N. Rosen- 
bluth, A. H. Teller, and E. Teller (1953). Equa- 
tion of state calculations by fast computing 
machines. Journal of Chemical Physics 21(6), 
1087-1092. 


Metropolis, N. and S. Ulam (1949). The Monte 
Carlo method. Journal of the American Statisti- 
cal Association 44(247), 335-341. 


Mika, S., G. Ratsch, J. Weston, and B. Scholkopf 
(1999). Fisher discriminant analysis with ker- 
nels. In Y. H. Hu, J. Larsen, E. Wilson, and 
S. Douglas (Eds.), Neural Networks for Signal 
Processing IX, pp. 41-48. IEEE. 


Minka, T. (2001a). Expectation propagation for ap- 
proximate Bayesian inference. In J. Breese and 
D. Koller (Eds.), Proceedings of the Seventeenth 
Conference on Uncertainty in Artificial Intelli- 
gence, pp. 362-369. Morgan Kaufmann. 

Minka, T. (2001b). A family of approximate al- 
gorithms for Bayesian inference. Ph. D. thesis, 
MIT. 


Minka, T. (2004). Power EP. Technical Report 
MSR-TR-2004-149, Microsoft Research Cam- 
bridge. 


Minka, T. (2005). Divergence measures and mes- 
sage passing. Technical Report MSR-TR-2005- 
173, Microsoft Research Cambridge. 


Minka, T. P. (2001c). Automatic choice of dimen- 
sionality for PCA. In T. K. Leen, T. G. Diet- 
terich, and V. Tresp (Eds.), Advances in Neural 


Information Processing Systems, Volume 13, pp. 
598-604. MIT Press. 


Minsky, M. L. and S. A. Papert (1969). Perceptrons. 
MIT Press. Expanded edition 1990. 


Miskin, J. W. and D. J. C. MacKay (2001). Ensem- 
ble learning for blind source separation. In S. J. 
Roberts and R. M. Everson (Eds.), Independent 
Component Analysis: Principles and Practice. 
Cambridge University Press. 


Møller, M. (1993). Efficient Training of Feed- 
Forward Neural Networks. Ph. D. thesis, Aarhus 
University, Denmark. 


Moody, J. and C. J. Darken (1989). Fast learning in 
networks of locally-tuned processing units. Neu- 
ral Computation 1(2), 281-294. 


Moore, A. W. (2000). The anchors hierarch: us- 
ing the triangle inequality to survive high dimen- 
sional data. In Proceedings of the Twelfth Con- 
ference on Uncertainty in Artificial Intelligence, 
pp. 397-405. 


Muller, K. R., S. Mika, G. Ratsch, K. Tsuda, and 
B. Scholkopf (2001). An introduction to kernel- 
based learning algorithms. [EEE Transactions on 
Neural Networks 12(2), 181-202. 


Miller, P. and F. A. Quintana (2004). Nonparametric 
Bayesian data analysis. Statistical Science 19(1), 
95-110. 


Nabney, I. T. (2002). Netlab: Algorithms for Pattern 
Recognition. Springer. 


Nadaraya, E. A. (1964). On estimating regression. 
Theory of Probability and its Applications 9(1), 
141-142. 


Nag, R., K. Wong, and F. Fallside (1986). Script 
recognition using hidden markov models. In 
ICASSP86, pp. 2071-2074. TEEE. 


Neal, R. M. (1993). Probabilistic inference using 
Markov chain Monte Carlo methods. Technical 
Report CRG-TR-93-1, Department of Computer 
Science, University of Toronto, Canada. 


Neal, R. M. (1996). Bayesian Learning for Neural 
Networks. Springer. Lecture Notes in Statistics 
118. 


Neal, R. M. (1997). Monte Carlo implementation of 
Gaussian process models for Bayesian regression 
and classification. Technical Report 9702, De- 
partment of Computer Statistics, University of 
Toronto. 


Neal, R. M. (1999). Suppressing random walks in 
Markov chain Monte Carlo using ordered over- 
relaxation. In M. I. Jordan (Ed.), Learning in 
Graphical Models, pp. 205-228. MIT Press. 


Neal, R. M. (2000). Markov chain sampling for 
Dirichlet process mixture models. Journal of 
Computational and Graphical Statistics 9, 249— 
265. 


Neal, R. M. (2003). Slice sampling. Annals of Statis- 
tics 31, 705-767. 


Neal, R. M. and G. E. Hinton (1999). A new view of 
the EM algorithm that justifies incremental and 
other variants. In M. I. Jordan (Ed.), Learning in 
Graphical Models, pp. 355-368. MIT Press. 


Nelder, J. A. and R. W. M. Wedderburn (1972). Gen- 
eralized linear models. Journal of the Royal Sta- 
tistical Society, A 135, 370-384. 


Nilsson, N. J. (1965). Learning Machines. McGraw- 
Hill. Reprinted as The Mathematical Founda- 
tions of Learning Machines, Morgan Kaufmann, 
(1990). 

Nocedal, J. and S. J. Wright (1999). Numerical Op- 
timization. Springer. 


Nowlan, S. J. and G. E. Hinton (1992). Simplifying 
neural networks by soft weight sharing. Neural 
Computation 4(4), 473-493. 


REFERENCES 723 


Ogden, R. T. (1997). Essential Wavelets for Statisti- 
cal Applications and Data Analysis. Birkhauser. 


Opper, M. and O. Winther (1999). A Bayesian ap- 
proach to on-line learning. In D. Saad (Ed.), On- 
Line Learning in Neural Networks, pp. 363-378. 
Cambridge University Press. 


Opper, M. and O. Winther (2000a). Gaussian 
processes and SVM: mean field theory and 
leave-one-out. In A. J. Smola, P. L. Bartlett, 
B. Scholkopf, and D. Shuurmans (Eds.), Ad- 
vances in Large Margin Classifiers, pp. 311-326. 
MIT Press. 


Opper, M. and O. Winther (2000b). Gaussian 
processes for classification. Neural Computa- 
tion 12(11), 2655-2684. 


Osuna, E., R. Freund, and F. Girosi (1996). Support 
vector machines: training and applications. A.I. 
Memo AIM-1602, MIT. 


Papoulis, A. (1984). Probability, Random Variables, 
and Stochastic Processes (Second ed.). McGraw- 
Hill. 


Parisi, G. (1988). Statistical Field Theory. Addison- 
Wesley. 


Pearl, J. (1988). Probabilistic Reasoning in Intelli- 
gent Systems. Morgan Kaufmann. 


Pearlmutter, B. A. (1994). Fast exact multiplication 
by the Hessian. Neural Computation 6(1), 147- 
160. 


Pearlmutter, B. A. and L. C. Parra (1997). Maximum 
likelihood source separation: a context-sensitive 
generalization of ICA. In M. C. Mozer, M. I. Jor- 
dan, and T. Petsche (Eds.), Advances in Neural 
Information Processing Systems, Volume 9, pp. 
613-619. MIT Press. 


Pearson, K. (1901). On lines and planes of closest fit 
to systems of points in space. The London, Edin- 
burgh and Dublin Philosophical Magazine and 
Journal of Science, Sixth Series 2, 559-572. 


Platt, J. C. (1999). Fast training of support vector 
machines using sequential minimal optimization. 
In B. Scholkopf, C. J. C. Burges, and A. J. Smola 
(Eds.), Advances in Kernel Methods — Support 
Vector Learning, pp. 185-208. MIT Press. 


724 REFERENCES 


Platt, J. C. (2000). Probabilities for SV machines. 
In A. J. Smola, P. L. Bartlett, B. Scholkopf, and 
D. Shuurmans (Eds.), Advances in Large Margin 
Classifiers, pp. 61-73. MIT Press. 


Platt, J. C., N. Cristianini, and J. Shawe-Taylor 
(2000). Large margin DAGs for multiclass clas- 
sification. In S. A. Solla, T. K. Leen, and K. R. 
Muller (Eds.), Advances in Neural Information 
Processing Systems, Volume 12, pp. 547-553. 
MIT Press. 


Poggio, T. and F. Girosi (1990). Networks for ap- 
proximation and learning. Proceedings of the 
IEEE 78(9), 1481-1497. 


Powell, M. J. D. (1987). Radial basis functions for 
multivariable interpolation: a review. In J. C. 
Mason and M. G. Cox (Eds.), Algorithms for 
Approximation, pp. 143-167. Oxford University 
Press. 


Press, W. H., S. A. Teukolsky, W. T. Vetterling, and 
B. P. Flannery (1992). Numerical Recipes in C: 
The Art of Scientific Computing (Second ed.). 
Cambridge University Press. 


Qazaz, C. S., C. K. I. Williams, and C. M. Bishop 
(1997). An upper bound on the Bayesian error 
bars for generalized linear regression. In S. W. 
Ellacott, J. C. Mason, and I. J. Anderson (Eds.), 
Mathematics of Neural Networks: Models, Algo- 
rithms and Applications, pp. 295-299. Kluwer. 


Quinlan, J. R. (1986). Induction of decision trees. 
Machine Learning 1(1), 81—106. 


Quinlan, J. R. (1993). C4.5: Programs for Machine 
Learning. Morgan Kaufmann. 


Rabiner, L. and B. H. Juang (1993). Fundamentals 
of Speech Recognition. Prentice Hall. 


Rabiner, L. R. (1989). A tutorial on hidden Markov 
models and selected applications in speech 
recognition. Proceedings of the IEEE 77(2), 
257-285. 


Ramasubramanian, V. and K. K. Paliwal (1990). A 
generalized optimization of the k-d tree for fast 
nearest-neighbour search. In Proceedings Fourth 
IEEE Region 10 International Conference (TEN- 
CON’S89), pp. 565-568. 


Ramsey, F. (1931). Truth and probability. In 
R. Braithwaite (Ed.), The Foundations of Math- 
ematics and other Logical Essays. Humanities 
Press. 


Rao, C. R. and S. K. Mitra (1971). Generalized In- 
verse of Matrices and Its Applications. Wiley. 


Rasmussen, C. E. (1996). Evaluation of Gaussian 
Processes and Other Methods for Non-Linear 
Regression. Ph. D. thesis, University of Toronto. 


Rasmussen, C. E. and J. Quifionero-Candela (2005). 
Healing the relevance vector machine by aug- 
mentation. In L. D. Raedt and S. Wrobel (Eds.), 
Proceedings of the 22nd International Confer- 
ence on Machine Learning, pp. 689-696. 


Rasmussen, C. E. and C. K. I. Williams (2006). 
Gaussian Processes for Machine Learning. MIT 
Press. 


Rauch, H. E., F. Tung, and C. T. Striebel (1965). 
Maximum likelihood estimates of linear dynam- 
ical systems. AZAA Journal 3, 1445-1450. 


Ricotti, L. P., S. Ragazzini, and G. Martinelli (1988). 
Learning of word stress in a sub-optimal second 
order backpropagation neural network. In Pro- 
ceedings of the IEEE International Conference 
on Neural Networks, Volume 1, pp. 355-361. 
IEEE. 


Ripley, B. D. (1996). Pattern Recognition and Neu- 
ral Networks. Cambridge University Press. 


Robbins, H. and S. Monro (1951). A stochastic 
approximation method. Annals of Mathematical 
Statistics 22, 400—407. 


Robert, C. P. and G. Casella (1999). Monte Carlo 
Statistical Methods. Springer. 


Rockafellar, R. (1972). Convex Analysis. Princeton 
University Press. 


Rosenblatt, F. (1962). Principles of Neurodynam- 
ics: Perceptrons and the Theory of Brain Mech- 
anisms. Spartan. 


Roth, V. and V. Steinhage (2000). Nonlinear discrim- 
inant analysis using kernel functions. In S. A. 


Solla, T. K. Leen, and K. R. Muller (Eds.), Ad- 
vances in Neural Information Processing Sys- 
tems, Volume 12. MIT Press. 


Roweis, S. (1998). EM algorithms for PCA and 
SPCA. In M. I. Jordan, M. J. Kearns, and S. A. 
Solla (Eds.), Advances in Neural Information 
Processing Systems, Volume 10, pp. 626-632. 
MIT Press. 


Roweis, S. and Z. Ghahramani (1999). A unifying 
review of linear Gaussian models. Neural Com- 
putation 11(2), 305-345. 


Roweis, S. and L. Saul (2000, December). Nonlinear 
dimensionality reduction by locally linear em- 
bedding. Science 290, 2323-2326. 


Rubin, D. B. (1983). Iteratively reweighted least 
squares. In Encyclopedia of Statistical Sciences, 
Volume 4, pp. 272-275. Wiley. 


Rubin, D. B. and D. T. Thayer (1982). EM al- 
gorithms for ML factor analysis. Psychome- 
trika 47(1), 69-76. 


Rumelhart, D. E., G. E. Hinton, and R. J. Williams 
(1986). Learning internal representations by er- 
ror propagation. In D. E. Rumelhart, J. L. Mc- 
Clelland, and the PDP Research Group (Eds.), 
Parallel Distributed Processing: Explorations 
in the Microstructure of Cognition, Volume 1: 
Foundations, pp. 318-362. MIT Press. Reprinted 
in Anderson and Rosenfeld (1988). 


Rumelhart, D. E., J. L. McClelland, and the PDP Re- 
search Group (Eds.) (1986). Parallel Distributed 
Processing: Explorations in the Microstruc- 
ture of Cognition, Volume 1: Foundations. MIT 
Press. 


Sagan, H. (1969). Introduction to the Calculus of 
Variations. Dover. 


Savage, L. J. (1961). The subjective basis of sta- 
tistical practice. Technical report, Department of 
Statistics, University of Michigan, Ann Arbor. 


Scholkopf, B., J. Platt, J. Shawe-Taylor, A. Smola, 
and R. C. Williamson (2001). Estimating the sup- 
port of a high-dimensional distribution. Neural 
Computation 13(7), 1433-1471. 


REFERENCES 725 


Scholkopf, B., A. Smola, and K.-R. Muller (1998). 
Nonlinear component analysis as a kernel 
eigenvalue problem. Neural Computation 10(5), 
1299-1319. 


Scholkopf, B., A. Smola, R. C. Williamson, and P. L. 
Bartlett (2000). New support vector algorithms. 
Neural Computation 12(5), 1207-1245. 


Scholkopf, B. and A. J. Smola (2002). Learning with 
Kernels. MIT Press. 


Schwarz, G. (1978). Estimating the dimension of a 
model. Annals of Statistics 6, 461—464. 


Schwarz, H. R. (1988). Finite element methods. Aca- 
demic Press. 


Seeger, M. (2003). Bayesian Gaussian Process Mod- 
els: PAC-Bayesian Generalization Error Bounds 
and Sparse Approximations. Ph. D. thesis, Uni- 
versity of Edinburg. 


Seeger, M., C. K. I. Williams, and N. Lawrence 
(2003). Fast forward selection to speed up sparse 
Gaussian processes. In C. M. Bishop and B. Frey 
(Eds.), Proceedings Ninth International Work- 
shop on Artificial Intelligence and Statistics, Key 
West, Florida. 


Shachter, R. D. and M. Peot (1990). Simulation ap- 
proaches to general probabilistic inference on be- 
lief networks. In P. P. Bonissone, M. Henrion, 
L. N. Kanal, and J. F. Lemmer (Eds.), Uncer- 
tainty in Artificial Intelligence, Volume 5. Else- 
vier. 


Shannon, C. E. (1948). A mathematical theory of 
communication. The Bell System Technical Jour- 
nal 27(3), 379-423 and 623-656. 


Shawe-Taylor, J. and N. Cristianini (2004). Kernel 
Methods for Pattern Analysis. Cambridge Uni- 
versity Press. 

Sietsma, J. and R. J. F Dow (1991). Creating artifi- 


cial neural networks that generalize. Neural Net- 
works 4(1), 67-79. 


Simard, P., Y. Le Cun, and J. Denker (1993). Effi- 
cient pattern recognition using a new transforma- 
tion distance. In S. J. Hanson, J. D. Cowan, and 


726 REFERENCES 


C. L. Giles (Eds.), Advances in Neural Informa- 
tion Processing Systems, Volume 5, pp. 50-58. 
Morgan Kaufmann. 


Simard, P., B. Victorri, Y. Le Cun, and J. Denker 
(1992). Tangent prop — a formalism for specify- 
ing selected invariances in an adaptive network. 
In J. E. Moody, S. J. Hanson, and R. P. Lippmann 
(Eds.), Advances in Neural Information Process- 
ing Systems, Volume 4, pp. 895-903. Morgan 
Kaufmann. 


Simard, P. Y., D. Steinkraus, and J. Platt (2003). 
Best practice for convolutional neural networks 
applied to visual document analysis. In Pro- 
ceedings International Conference on Document 
Analysis and Recognition (ICDAR), pp. 958- 
962. IEEE Computer Society. 


Sirovich, L. (1987). Turbulence and the dynamics 
of coherent structures. Quarterly Applied Math- 
ematics 45(3), 561-590. 


Smola, A. J. and P. Bartlett (2001). Sparse greedy 
Gaussian process regression. In T. K. Leen, T. G. 
Dietterich, and V. Tresp (Eds.), Advances in Neu- 
ral Information Processing Systems, Volume 13, 
pp. 619-625. MIT Press. 


Spiegelhalter, D. and S. Lauritzen (1990). Sequential 
updating of conditional probabilities on directed 
graphical structures. Networks 20, 579-605. 


Stinchecombe, M. and H. White (1989). Universal 
approximation using feed-forward networks with 
non-sigmoid hidden layer activation functions. In 
International Joint Conference on Neural Net- 
works, Volume 1, pp. 613-618. IEEE. 


Stone, J. V. (2004). Independent Component Analy- 
sis: A Tutorial Introduction. MIT Press. 


Sung, K. K. and T. Poggio (1994). Example-based 
learning for view-based human face detection. 
A.I. Memo 1521, MIT. 


Sutton, R. S. and A. G. Barto (1998). Reinforcement 
Learning: An Introduction. MIT Press. 


Svensén, M. and C. M. Bishop (2004). Ro- 
bust Bayesian mixture modelling. Neurocomput- 
ing 64, 235-252. 


Tarassenko, L. (1995). Novelty detection for the 
identification of masses in mamograms. In Pro- 
ceedings Fourth IEE International Conference 
on Artificial Neural Networks, Volume 4, pp. 
442-447. IEE. 


Tax, D. and R. Duin (1999). Data domain descrip- 
tion by support vectors. In M. Verleysen (Ed.), 
Proceedings European Symposium on Artificial 
Neural Networks, ESANN, pp. 251-256. D. Facto 
Press. 


Teh, Y. W., M. I. Jordan, M. J. Beal, and D. M. Blei 
(2006). Hierarchical Dirichlet processes. Journal 
of the Americal Statistical Association. to appeat. 


Tenenbaum, J. B., V. de Silva, and J. C. Langford 
(2000, December). A global framework for non- 
linear dimensionality reduction. Science 290, 
2319-2323. 


Tesauro, G. (1994). TD-Gammon, a self-teaching 
backgammon program, achieves master-level 
play. Neural Computation 6(2), 215-219. 


Thiesson, B., D. M. Chickering, D. Heckerman, and 
C. Meek (2004). ARMA time-series modelling 
with graphical models. In M. Chickering and 
J. Halpern (Eds.), Proceedings of the Twentieth 
Conference on Uncertainty in Artificial Intelli- 
gence, Banff, Canada, pp. 552-560. AUAI Press. 


Tibshirani, R. (1996). Regression shrinkage and se- 
lection via the lasso. Journal of the Royal Statis- 
tical Society, B 58, 267-288. 


Tierney, L. (1994). Markov chains for exploring pos- 
terior distributions. Annals of Statistics 22(4), 
1701-1762. 


Tikhonov, A. N. and V. Y. Arsenin (1977). Solutions 
of Ill-Posed Problems. V. H. Winston. 


Tino, P. and I. T. Nabney (2002). Hierarchical 
GTM: constructing localized non-linear projec- 
tion manifolds in a principled way. IEEE Trans- 


actions on Pattern Analysis and Machine Intelli- 
gence 24(5), 639-656. 


Tino, P., I. T. Nabney, and Y. Sun (2001). Us- 
ing directional curvatures to visualize folding 
patterns of the GTM projection manifolds. In 


G. Dorffner, H. Bischof, and K. Hornik (Eds.), 
Artificial Neural Networks — ICANN 2001, pp. 
421-428. Springer. 


Tipping, M. E. (1999). Probabilistic visualisation of 
high-dimensional binary data. In M. S. Kearns, 
S. A. Solla, and D. A. Cohn (Eds.), Advances 
in Neural Information Processing Systems, Vol- 
ume 11, pp. 592-598. MIT Press. 


Tipping, M. E. (2001). Sparse Bayesian learning and 
the relevance vector machine. Journal of Ma- 
chine Learning Research 1, 211-244. 


Tipping, M. E. and C. M. Bishop (1997). Probabilis- 
tic principal component analysis. Technical Re- 
port NCRG/97/010, Neural Computing Research 
Group, Aston University. 


Tipping, M. E. and C. M. Bishop (1999a). Mixtures 
of probabilistic principal component analyzers. 
Neural Computation 11(2), 443-482. 


Tipping, M. E. and C. M. Bishop (1999b). Prob- 
abilistic principal component analysis. Journal 
of the Royal Statistical Society, Series B 21(3), 
611-622. 


Tipping, M. E. and A. Faul (2003). Fast marginal 
likelihood maximization for sparse Bayesian 
models. In C. M. Bishop and B. Frey (Eds.), 
Proceedings Ninth International Workshop on 
Artificial Intelligence and Statistics, Key West, 
Florida. 


Tong, S. and D. Koller (2000). Restricted Bayes op- 
timal classifiers. In Proceedings 17th National 
Conference on Artificial Intelligence, pp. 658- 
664. AAAI. 


Tresp, V. (2001). Scaling kernel-based systems to 
large data sets. Data Mining and Knowledge Dis- 
covery 5(3), 197-211. 


Uhlenbeck, G. E. and L. S. Ornstein (1930). On the 
theory of Brownian motion. Phys. Rev. 36, 823- 
841. 


Valiant, L. G. (1984). A theory of the learnable. 
Communications of the Association for Comput- 
ing Machinery 27, 1134-1142. 


REFERENCES 727 


Vapnik, V. N. (1982). Estimation of dependences 
based on empirical data. Springer. 


Vapnik, V. N. (1995). The nature of statistical learn- 
ing theory. Springer. 

Vapnik, V. N. (1998). Statistical learning theory. Wi- 
ley. 

Veropoulos, K., C. Campbell, and N. Cristianini 
(1999). Controlling the sensitivity of support 
vector machines. In Proceedings of the Interna- 


tional Joint Conference on Artificial Intelligence 
(IJCAI99), Workshop ML3, pp. 55-60. 


Vidakovic, B. (1999). Statistical Modelling by 
Wavelets. Wiley. 


Viola, P. and M. Jones (2004). Robust real-time face 
detection. International Journal of Computer Vi- 
sion 57(2), 137-154. 


Viterbi, A. J. (1967). Error bounds for convolu- 
tional codes and an asymptotically optimum de- 
coding algorithm. IEEE Transactions on Infor- 
mation Theory IT-13, 260-267. 


Viterbi, A. J. and J. K. Omura (1979). Principles of 
Digital Communication and Coding. McGraw- 
Hill. 


Wahba, G. (1975). A comparison of GCV and GML 
for choosing the smoothing parameter in the gen- 
eralized spline smoothing problem. Numerical 
Mathematics 24, 383-393. 


Wainwright, M. J., T. S. Jaakkola, and A. S. Willsky 
(2005). A new class of upper bounds on the log 
partition function. IEEE Transactions on Infor- 
mation Theory 51, 2313-2335. 


Walker, A. M. (1969). On the asymptotic behaviour 
of posterior distributions. Journal of the Royal 
Statistical Society, B 31(1), 80-88. 


Walker, S. G., P. Damien, P. W. Laud, and A. F. M. 
Smith (1999). Bayesian nonparametric inference 
for random distributions and related functions 
(with discussion). Journal of the Royal Statisti- 
cal Society, B 61(3), 485-527. 


Watson, G. S. (1964). Smooth regression analysis. 
Sankhya: The Indian Journal of Statistics. Series 
A 26, 359-372. 


728 REFERENCES 


Webb, A. R. (1994). Functional approximation by Williams, O., A. Blake, and R. Cipolla (2005). 


feed-forward networks: a least-squares approach Sparse Bayesian learning for efficient visual 
to generalisation. IEEE Transactions on Neural tracking. IEEE Transactions on Pattern Analysis 
Networks 5(3), 363—371. and Machine Intelligence 27(8), 1292-1304. 
Weisstein, E. W. (1999). CRC Concise Encyclopedia Williams, P. M. (1996). Using neural networks to 
of Mathematics. Chapman and Hall, and CRC. model conditional multivariate densities. Neural 


Weston, J. and C. Watkins (1999). Multi-class sup- Computation 8(4), 843-854. 
port vector machines. In M. Verlysen (Ed.), Pro- Winn, J. and C. M. Bishop (2005). Variational mes- 
ceedings ESANN’99, Brussels. D-Facto Publica- sage passing. Journal of Machine Learning Re- 


tions. search 6, 661-694. 
Whittaker, J. (1990). Graphical Models in Applied Zarchan, P. and H. Musoff (2005). Fundamentals of 
Multivariate Statistics. Wiley. Kalman Filtering: A Practical Approach (Sec- 
ond ed.). AIAA. 


Widrow, B. and M. E. Hoff (1960). Adaptive 
switching circuits. In JRE WESCON Convention 
Record, Volume 4, pp. 96-104. Reprinted in An- 
derson and Rosenfeld (1988). 


Widrow, B. and M. A. Lehr (1990). 30 years of adap- 
tive neural networks: perceptron, madeline, and 
backpropagation. Proceedings of the IEEE 78(9), 
1415-1442. 


Wiegerinck, W. and T. Heskes (2003). Fractional 
belief propagation. In S. Becker, S. Thrun, and 
K. Obermayer (Eds.), Advances in Neural Infor- 
mation Processing Systems, Volume 15, pp. 455- 
462. MIT Press. 


Williams, C. K. I. (1998). Computation with infi- 
nite neural networks. Neural Computation 10(5), 
1203-1216. 


Williams, C. K. I. (1999). Prediction with Gaussian 
processes: from linear regression to linear pre- 
diction and beyond. In M. I. Jordan (Ed.), Learn- 
ing in Graphical Models, pp. 599-621. MIT 
Press. 


Williams, C. K. I. and D. Barber (1998). Bayesian 
classification with Gaussian processes. [EEE 
Transactions on Pattern Analysis and Machine 
Intelligence 20, 1342-1351. 


Williams, C. K. I. and M. Seeger (2001). Using the 
Nystrom method to speed up kernel machines. In 
T. K. Leen, T. G. Dietterich, and V. Tresp (Eds.), 
Advances in Neural Information Processing Sys- 
tems, Volume 13, pp. 682—688. MIT Press. 


Index 


INDEX 729 


Page numbers in bold indicate the primary source of information for the corresponding topic. 


1-of-K coding scheme, 424 


acceptance criterion, 538, 541, 544 

activation function, 180, 213, 227 

active constraint, 328, 709 

AdaBoost, 657, 658 

adaline, 196 

adaptive rejection sampling, 530 

ADF, see assumed density filtering 

AIC, see Akaike information criterion 

Akaike information criterion, 33, 217 

a family of divergences, 469 

a recursion, 620 

ancestral sampling, 365, 525, 613 

annular flow, 679 

AR model, see autoregressive model 

arc, 360 

ARD, see automatic relevance determination 

ARMA, see autoregressive moving average 

assumed density filtering, 510 

autoassociative networks, 592 

automatic relevance determination, 259, 312, 349, 
485, 582 

autoregressive hidden Markov model, 632 

autoregressive model, 609 

autoregressive moving average, 304 


back-tracking, 415, 630 


backgammon, 3 

backpropagation, 241 

bagging, 656 

basis function, 138, 172, 204, 227 

batch training, 240 

Baum-Welch algorithm, 618 

Bayes’ theorem, 15 

Bayes, Thomas, 21 

Bayesian analysis, vii, 9, 21 
hierarchical, 372 
model averaging, 654 

Bayesian information criterion, 33, 216 

Bayesian model comparison, 161, 473, 483 

Bayesian network, 360 

Bayesian probability, 21 

belief propagation, 403 

Bernoulli distribution, 69, 113, 685 
mixture model, 444 

Bernoulli, Jacob, 69 

beta distribution, 71, 686 

beta recursion, 621 

between-class covariance, 189 

bias, 27, 149 

bias parameter, 138, 181, 227, 346 

bias-variance trade-off, 147 

BIC, see Bayesian information criterion 

binary entropy, 495 

binomial distribution, 70, 686 
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biological sequence, 610 concentration parameter, 108, 693 
bipartite graph, 401 condensation algorithm, 646 
bits, 49 conditional entropy, 55 
blind source separation, 591 conditional expectation, 20 
blocked path, 374, 378, 384 conditional independence, 46, 372, 383 
Boltzmann distribution, 387 conditional mixture model, see mixture model 
Boltzmann, Ludwig Eduard, 53 conditional probability, 14 
Boolean logic, 21 conjugate prior, 68, 98, 117, 490 
boosting, 657 convex duality, 494 
bootstrap, 23, 656 convex function, 55, 493 
bootstrap filter, 646 convolutional neural network, 267 
box constraints, 333, 342 correlation matrix, 567 
Box-Muller method, 527 cost function, 41 
covariance, 20 
C4.5, 663 between-class, 189 
calculus of variations, 462 within-class, 189 
canonical correlation analysis, 565 covariance matrix 
canonical link function, 212 diagonal, 84 
CART, see classification and regression trees isotropic, 84 
Cauchy distribution, 527, 529, 692 partitioned, 85, 307 
causality, 366 positive definite, 308 
CCA, see canonical correlation analysis Cox’s axioms, 21 
central differences, 246 credit assignment, 3 
central limit theorem, 78 cross-entropy error function, 206, 209, 235, 631, 
chain graph, 393 666 
chaining, 555 cross-validation, 32, 161 
Chapman-Kolmogorov equations, 397 cumulative distribution function, 18 
child node, 361 curse of dimensionality, 33, 36 
Cholesky decomposition, 528 curve fitting, 4 
chunking, 335 
circular normal, see von Mises distribution D map, see dependency map 
classical probability, 21 d-separation, 373, 378, 443 
classification, 3 DAG, see directed acyclic graph 
classification and regression trees, 663 DAGSVM, 339 
clique, 385 data augmentation, 537 
clustering, 3 data compression, 429 
clutter problem, 511 decision boundary, 39, 179 
co-parents, 383, 492 decision region, 39, 179 
code-book vectors, 429 decision surface, see decision boundary 
combining models, 45, 653 decision theory, 38 
committee, 655 decision tree, 654, 663, 673 
complete data set, 440 decomposition methods, 335 
completing the square, 86 degrees of freedom, 559 
computational learning theory, 326, 344 degrees-of-freedom parameter, 102, 693 


concave function, 56 density estimation, 3, 67 


density network, 597 

dependency map, 392 

descendant node, 376 

design matrix, 142, 347 
differential entropy, 53 

digamma function, 687 

directed acyclic graph, 362 
directed cycle, 362 

directed factorization, 381 
Dirichlet distribution, 76, 687 
Dirichlet, Lejeune, 77 
discriminant function, 43, 180, 181 
discriminative model, 43, 203 
distortion measure, 424 
distributive law of multiplication, 396 
DNA, 610 

document retrieval, 299 

dual representation, 293, 329 
dual-energy gamma densitometry, 678 
dynamic programming, 411 
dynamical system, 548 


E step, see expectation step 
early stopping, 259 
ECM, see expectation conditional maximization 
edge, 360 
effective number of observations, 72, 101 
effective number of parameters, 9, 170, 281 
elliptical K-means, 444 
EM, see expectation maximization 
emission probability, 611 
empirical Bayes, see evidence approximation 
energy function, 387 
entropy, 49 
conditional, 55 
differential, 53 
relative, 55 
EP, see expectation propagation 
e-tube, 341 
€-insensitive error function, 340 
equality constraint, 709 
equivalent kernel, 159, 301 
erf function, 211 
error backpropagation, see backpropagation 
error function, 5, 23 
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error-correcting output codes, 339 
Euler, Leonhard, 465 
Euler-Lagrange equations, 705 
evidence approximation, 165, 347, 581 
evidence function, 161 
expectation, 19 
expectation conditional maximization, 454 
expectation maximization, 113, 423, 440 
Gaussian mixture, 435 
generalized, 454 
sampling methods, 536 
expectation propagation, 315, 468, 505 
expectation step, 437 
explaining away, 378 
exploitation, 3 
exploration, 3 
exponential distribution, 526, 688 
exponential family, 68, 113, 202, 490 
extensive variables, 490 


face detection, 2 
face tracking, 355 
factor analysis, 583 

mixture model, 595 
factor graph, 360, 399, 625 
factor loading, 584 
factorial hidden Markov model, 633 
factorized distribution, 464, 476 
feature extraction, 2 
feature map, 268 
feature space, 292, 586 
Fisher information matrix, 298 
Fisher kernel, 298 
Fisher’s linear discriminant, 186 
flooding schedule, 417 
forward kinematics, 272 
forward problem, 272 
forward propagation, 228, 243 
forward-backward algorithm, 618 
fractional belief propagation, 517 
frequentist probability, 21 
fuel system, 376 
function interpolation, 299 
functional, 462, 703 

derivative, 463 
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gamma densitometry, 678 undirected, 360 
gamma distribution, 529, 688 Green’s function, 299 
gamma function, 71 GTM, see generative topographic mapping 
gating function, 672 
Gauss, Carl Friedrich, 79 Hamilton, William Rowan, 549 
Gaussian, 24, 78, 688 Hamiltonian dynamics, 548 
conditional, 85, 93 Hamiltonian function, 549 
marginal, 88, 93 Hammersley-Clifford theorem, 387 
maximum likelihood, 93 handwriting recognition, 1, 610, 614 
mixture, 110, 270, 273, 430 handwritten digit, 565, 614, 677 
sequential estimation, 94 head-to-head path, 376 
sufficient statistics, 93 head-to-tail path, 375 
wrapped, 110 Heaviside step function, 206 
Gaussian kernel, 296 Hellinger distance, 470 
Gaussian process, 160, 303 Hessian matrix, 167, 215, 217, 238, 249 
Gaussian random field, 305 diagonal approximation, 250 
Gaussian-gamma distribution, 101, 690 exact evaluation, 253 
Gaussian-Wishart distribution, 102, 475, 478, 690 fast multiplication, 254 
GEM, see expectation maximization, generalized finite differences, 252 
generalization, 2 inverse, 252 
generalized linear model, 180, 213 outer product approximation, 251 
generalized maximum likelihood, see evidence ap- _heteroscedastic, 273, 311 
proximation hidden Markov model, 297, 610 
generative model, 43, 196, 297, 365, 572, 631 autoregressive, 632 
generative topographic mapping, 597 factorial, 633 
directional curvature, 599 forward-backward algorithm, 618 
magnification factor, 599 input-output, 633 
geodesic distance, 596 left-to-right, 613 
Gibbs sampling, 542 maximum likelihood, 615 
blocking, 546 scaling factor, 627 
Gibbs, Josiah Willard, 543 sum-product algorithm, 625 
Gini index, 666 switching, 644 
global minimum, 237 variational inference, 625 
gradient descent, 240 hidden unit, 227 
Gram matrix, 293 hidden variable, 84, 364, 430, 559 
graph-cut algorithm, 390 hierarchical Bayesian model, 372 
graphical model, 359 hierarchical mixture of experts, 673 
bipartite, 401 hinge error function, 337 
directed, 360 Hinton diagram, 584 
factorization, 362, 384 histogram density estimation, 120 
fully connected, 361 HME, see hierarchical mixture of experts 
inference, 393 hold-out set, 11 
tree, 398 homogeneous flow, 679 
treewidth, 417 homogeneous kernel, 292 


triangulated, 416 homogeneous Markov chain, 540, 608 


Hooke’s law, 580 

hybrid Monte Carlo, 548 

hyperparameter, 71, 280, 311, 346, 372, 502 
hyperprior, 372 


I map, see independence map 

i.i.d., see independent identically distributed 
ICA, see independent component analysis 
ICM, see iterated conditional modes 
ID3, 663 

identifiability, 435 

image de-noising, 387 

importance sampling, 525, 532 
importance weights, 533 

improper prior, 118, 259, 472 
imputation step, 537 
imputation-posterior algorithm, 537 
inactive constraint, 328, 709 

incomplete data set, 440 

independence map, 392 

independent component analysis, 591 
independent factor analysis, 592 
independent identically distributed, 26, 379 
independent variables, 17 

independent, identically distributed, 605 
induced factorization, 485 

inequality constraint, 709 

inference, 38, 42 

information criterion, 33 

information geometry, 298 

information theory, 48 

input-output hidden Markov model, 633 
intensive variables, 490 

intrinsic dimensionality, 559 

invariance, 261 

inverse gamma distribution, 101 

inverse kinematics, 272 

inverse problem, 272 

inverse Wishart distribution, 102 


IP algorithm, see imputation-posterior algorithm 


IRLS, see iterative reweighted least squares 
Ising model, 389 

isomap, 596 

isometric feature map, 596 

iterated conditional modes, 389, 415 
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iterative reweighted least squares, 207, 210, 316, 
354, 672 


Jacobian matrix, 247, 264 
Jensen’s inequality, 56 

join tree, 416 

junction tree algorithm, 392, 416 


K nearest neighbours, 125 
K-means clustering algorithm, 424, 443 
K-medoids algorithm, 428 
Kalman filter, 304, 637 
extended, 644 
Kalman gain matrix, 639 
Kalman smoother, 637 
Karhunen-Loéve transform, 561 
Karush-Kuhn-Tucker conditions, 330, 333, 342, 
710 
kernel density estimator, 122, 326 
kernel function, 123, 292, 294 
Fisher, 298 
Gaussian, 296 
homogeneous, 292 
nonvectorial inputs, 297 
stationary, 292 
kernel PCA, 586 
kernel regression, 300, 302 
kernel substitution, 292 
kernel trick, 292 
kinetic energy, 549 
KKT, see Karush-Kuhn-Tucker conditions 
KL divergence, see Kullback-Leibler divergence 
kriging, see Gaussian process 
Kullback-Leibler divergence, 55, 451, 468, 505 


Lagrange multiplier, 707 
Lagrange, Joseph-Louis, 329 
Lagrangian, 328, 332, 341, 708 
laminar flow, 678 

Laplace approximation, 213, 217, 278, 315, 354 
Laplace, Pierre-Simon, 24 

large margin, see margin 

lasso, 145 

latent class analysis, 444 

latent trait model, 597 

latent variable, 84, 364, 430, 559 
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lattice diagram, 414, 611, 621, 629 
LDS, see linear dynamical system 
leapfrog discretization, 551 
learning, 2 
learning rate parameter, 240 
least-mean-squares algorithm, 144 
leave-one-out, 33 
likelihood function, 22 
likelihood weighted sampling, 534 
linear discriminant, 181 
Fisher, 186 
linear dynamical system, 84, 635 
inference, 638 
linear independence, 696 
linear regression, 138 
EM, 448 
mixture model, 667 
variational, 486 
linear smoother, 159 
linear-Gaussian model, 87, 370 
linearly separable, 179 
link, 360 
link function, 180, 213 
Liouville’s Theorem, 550 
LLE, see locally linear embedding 
LMS algorithm, see least-mean-squares algorithm 
local minimum, 237 
local receptive field, 268 
locally linear embedding, 596 
location parameter, 118 
log odds, 197 
logic sampling, 525 
logistic regression, 205, 336 
Bayesian, 217, 498 
mixture model, 670 
multiclass, 209 
logistic sigmoid, 114, 139, 197, 205, 220, 227, 495 
logit function, 197 
loopy belief propagation, 417 
loss function, 41 
loss matrix, 41 
lossless data compression, 429 
lossy data compression, 429 
lower bound, 484 


M step, see maximization step 


machine learning, vii 
macrostate, 51 
Mahalanobis distance, 80 
manifold, 38, 590, 595, 681 
MAP, see maximum posterior 
margin, 326, 327, 502 
error, 334 
soft, 332 
marginal likelihood, 162, 165 
marginal probability, 14 
Markov blanket, 382, 384, 545 
Markov boundary, see Markov blanket 
Markov chain, 397, 539 
first order, 607 
homogeneous, 540, 608 
second order, 608 
Markov chain Monte Carlo, 537 
Markov model, 607 
homogeneous, 612 
Markov network, see Markov random field 
Markov random field, 84, 360, 383 
max-sum algorithm, 411, 629 
maximal clique, 385 
maximal spanning tree, 416 
maximization step, 437 
maximum likelihood, 9, 23, 26, 116 
Gaussian mixture, 432 
singularities, 480 
type 2, see evidence approximation 
maximum margin, see margin 
maximum posterior, 30, 441 
MCMC, see Markov chain Monte Carlo 
MDN, see mixture density network 
MDS, see multidimensional scaling 
mean, 24 
mean field theory, 465 
mean value theorem, 52 
measure theory, 19 
memory-based methods, 292 
message passing, 396 
pending message, 417 
schedule, 417 
variational, 491 
Metropolis algorithm, 538 
Metropolis-Hastings algorithm, 541 


microstate, 51 
minimum risk, 44 
Minkowski loss, 48 
missing at random, 441, 579 
missing data, 579 
mixing coefficient, 111 
mixture component, 111 
mixture density network, 272, 673 
mixture distribution, see mixture model 
mixture model, 162, 423 
conditional, 273, 666 
linear regression, 667 
logistic regression, 670 
symmetries, 483 
mixture of experts, 672 
mixture of Gaussians, 110, 270, 273, 430 
MLP, see multilayer perceptron 
MNIST data, 677 
model comparison, 6, 32, 161, 473, 483 
model evidence, 161 
model selection, 162 
moment matching, 506, 510 
momentum variable, 548 
Monte Carlo EM algorithm, 536 
Monte Carlo sampling, 24, 523 
Moore-Penrose pseudo-inverse, see pseudo-inverse 
moralization, 391, 401 
MRF, see Markov random field 
multidimensional scaling, 596 
multilayer perceptron, 226, 229 
multimodality, 272 
multinomial distribution, 76, 114, 690 
multiplicity, 51 
mutual information, 55, 57 


Nadaraya- Watson, see kernel regression 
naive Bayes model, 46, 380 
nats, 50 
natural language modelling, 610 
natural parameters, 113 
nearest-neighbour methods, 124 
neural network, 225 
convolutional, 267 
regularization, 256 
relation to Gaussian process, 319 
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Newton-Raphson, 207, 317 

node, 360 

noiseless coding theorem, 50 
nonidentifiability, 585 

noninformative prior, 23, 117 

nonparametric methods, 68, 120 

normal distribution, see Gaussian 

normal equations, 142 

normal-gamma distribution, 101, 691 

normal- Wishart distribution, 102, 475, 478, 691 
normalized exponential, see softmax function 
novelty detection, 44 

v-SVM, 334 


object recognition, 366 

observed variable, 364 

Occam factor, 217 

oil flow data, 34, 560, 568, 678 

Old Faithful data, 110, 479, 484, 681 
on-line learning, see sequential learning 
one-versus-one classifier, 183, 339 
one-versus-the-rest classifier, 182, 338 
ordered over-relaxation, 545 
Ornstein-Uhlenbeck process, 305 
orthogonal least squares, 301 

outlier, 44, 185, 212 

outliers, 103 

over-fitting, 6, 147, 434, 464 
over-relaxation, 544 


PAC learning, see probably approximately correct 
PAC-Bayesian framework, 345 
parameter shrinkage, 144 
parent node, 361 
particle filter, 645 
partition function, 386, 554 
Parzen estimator, see kernel density estimator 
Parzen window, 123 
pattern recognition, vii 
PCA, see principal component analysis 
pending message, 417 
perceptron, 192 
convergence theorem, 194 
hardware, 196 
perceptron criterion, 193 
perfect map, 392 
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periodic variable, 105 
phase space, 549 
photon noise, 680 
plate, 363 
polynomial curve fitting, 4, 362 
polytree, 399 
position variable, 548 
positive definite covariance, 81 
positive definite matrix, 701 
positive semidefinite covariance, 81 
positive semidefinite matrix, 701 
posterior probability, 17 
posterior step, 537 
potential energy, 549 
potential function, 386 
power EP, 517 
power method, 563 
precision matrix, 85 
precision parameter, 24 
predictive distribution, 30, 156 
preprocessing, 2 
principal component analysis, 561, 572, 593 
Bayesian, 580 
EM algorithm, 577 
Gibbs sampling, 583 
mixture distribution, 595 
physical analogy, 580 
principal curve, 595 
principal subspace, 561 
principal surface, 596 
prior, 17 
conjugate, 68, 98, 117, 490 
consistent, 257 
improper, 118, 259, 472 
noninformative, 23, 117 
probabilistic graphical model, see graphical model 
probabilistic PCA, 570 
probability, 12 
Bayesian, 21 
classical, 21 
density, 17 
frequentist, 21 
mass function, 19 
prior, 45 
product rule, 13, 14, 359 


sum tule, 13, 14, 359 

theory, 12 
probably approximately correct, 344 
probit function, 211, 219 
probit regression, 210 
product rule of probability, 13, 14, 359 
proposal distribution, 528, 532, 538 
protected conjugate gradients, 335 
protein sequence, 610 
pseudo-inverse, 142, 185 
pseudo-random numbers, 526 


quadratic discriminant, 199 
quality parameter, 351 


radial basis function, 292, 299 
Rauch-Tung-Striebel equations, 637 
regression, 3 
regression function, 47, 95 
regularization, 10 

Tikhonov, 267 
regularized least squares, 144 
reinforcement learning, 3 
reject option, 42, 45 
rejection sampling, 528 
relative entropy, 55 
relevance vector, 348 
relevance vector machine, 161, 345 
responsibility, 112, 432, 477 
ridge regression, 10 
RMS error, see root-mean-square error 
Robbins-Monro algorithm, 95 
robot arm, 272 
robustness, 103, 185 
root node, 399 
root-mean-square error, 6 
Rosenblatt, Frank, 193 
rotation invariance, 573, 585 
RTS equations, see Rauch-Tung-Striebel equations 
running intersection property, 416 
RVM, see relevance vector machine 


sample mean, 27 

sample variance, 27 
sampling-importance-resampling, 534 
scale invariance, 119, 261 


scale parameter, 119 
scaling factor, 627 
Schwarz criterion, see Bayesian information crite- 
rion 
self-organizing map, 598 
sequential data, 605 
sequential estimation, 94 
sequential gradient descent, 144, 240 
sequential learning, 73, 143 
sequential minimal optimization, 335 
serial message passing schedule, 417 
Shannon, Claude, 55 
shared parameters, 368 
shrinkage, 10 
Shur complement, 87 
sigmoid, see logistic sigmoid 
simplex, 76 
single-class support vector machine, 339 
singular value decomposition, 143 
sinusoidal data, 682 
SIR, see sampling-importance-resampling 
skip-layer connection, 229 
slack variable, 331 
slice sampling, 546 
SMO, see sequential minimal optimization 
smoother matrix, 159 
smoothing parameter, 122 
soft margin, 332 
soft weight sharing, 269 
softmax function, 115, 198, 236, 274, 356, 497 
SOM, see self-organizing map 
sparsity, 145, 347, 349, 582 
sparsity parameter, 351 
spectrogram, 606 
speech recognition, 605, 610 
sphereing, 568 
spline functions, 139 
standard deviation, 24 
standardizing, 425, 567 
state space model, 609 
switching, 644 
stationary kernel, 292 
Statistical bias, see bias 
statistical independence, see independent variables 
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statistical learning theory, see computational learn- 
ing theory, 326, 344 

steepest descent, 240 
Stirling’s approximation, 51 
stochastic, 5 
stochastic EM, 536 
stochastic gradient descent, 144, 240 
stochastic process, 305 
stratified flow, 678 
Student’s t-distribution, 102, 483, 691 
subsampling, 268 
sufficient statistics, 69, 75, 116 
sum rule of probability, 13, 14, 359 
sum-of-squares error, 5, 29, 184, 232, 662 
sum-product algorithm, 399, 402 

for hidden Markov model, 625 
supervised learning, 3 
support vector, 330 
support vector machine, 225 

for regression, 339 

multiclass, 338 
survival of the fittest, 646 
SVD, see singular value decomposition 
SVM, see support vector machine 
switching hidden Markov model, 644 
switching state space model, 644 
synthetic data sets, 682 


tail-to-tail path, 374 

tangent distance, 265 

tangent propagation, 262, 263 
tapped delay line, 609 

target vector, 2 

test set, 2, 32 

threshold parameter, 181 

tied parameters, 368 

Tikhonov regularization, 267 
time warping, 615 

tomography, 679 

training, 2 

training set, 2 

transition probability, 540, 610 
translation invariance, 118, 261 
tree-reweighted message passing, 517 
treewidth, 417 
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trellis diagram, see lattice diagram 

triangulated graph, 416 

type 2 maximum likelihood, see evidence approxi- 
mation 


undetermined multiplier, see Lagrange multiplier 
undirected graph, see Markov random field 
uniform distribution, 692 

uniform sampling, 534 

uniquenesses, 584 

unobserved variable, see latent variable 
unsupervised learning, 3 

utility function, 41 


validation set, 11, 32 

Vapnik-Chervonenkis dimension, 344 

variance, 20, 24, 149 

variational inference, 315, 462, 635 
for Gaussian mixture, 474 
for hidden Markov model, 625 
local, 493 

VC dimension, see Vapnik-Chervonenkis dimen- 

sion 

vector quantization, 429 

vertex, see node 

visualization, 3 

Viterbi algorithm, 415, 629 

von Mises distribution, 108, 693 


wavelets, 139 
weak learner, 657 
weight decay, 10, 144, 257 
weight parameter, 227 
weight sharing, 268 

soft, 269 
weight vector, 181 
weight-space symmetry, 231, 281 
weighted least squares, 668 
well-determined parameters, 170 
whitening, 299, 568 
Wishart distribution, 102, 693 
within-class covariance, 189 
Woodbury identity, 696 
wrapped distribution, 110 


Yellowstone National Park, 110, 681 


